SDMX metadata output

[brockfanning]

Fixes #232

This adds metadata to the existing SDMX output class. This is a quick-fix solution using a low-tech templating approach. Longer term it would be nice to get this functionality into the Python SDMX library, but this works for now. The metadata is placed in a 'meta' subfolder of the 'sdmx' folder.

New configuration

This adds some new parameters to the SdmxMlOutput class:

• msd: When in combination with constrain_meta below, can limit which fields are output.
• constrain_meta: See msd above.
• meta_ref_area: This sets the REF_AREA to use in the metadata output. If omitted, it will use the first REF_AREA value in the data (if available).
• meta_reporting_type: This sets the REPORTING_TYPE to use in the metadata output. If omitted, it will use the first REPORTING_TYPE value in the data (if available).

[LucyGwilliamAdmin]

@brockfanning is there a way to constrain to a specific schema e.g. MSD and not prose fields?

[brockfanning]

@LucyGwilliamAdmin Right now the constrain_meta parameter will limit to whatever is being used as the metadata schema. I guess we may need more control than that. For example, if the metadata schema includes "my_other_field", we may want a way to keep that field out of the SDMX output, right? I'm just wondering what would be a good parameter for that.

[brockfanning]

@LucyGwilliamAdmin I've added an msd parameter. If you pass in a path to an MSD, and also have constrain_meta set to true, then the metadata should be limited to whatever is in the MSD. If you have constrain_meta set to true, but you do not specify an MSD, it will assume the global MSD.

[brockfanning]

@LucyGwilliamAdmin This one also might be a good addition to 1.4.0 - so maybe we could merge and test during the beta testing?

[LucyGwilliamAdmin]

@brockfanning I'm just in the middle of giving this one a test - shouldn't be too long on it

[LucyGwilliamAdmin]

@brockfanning I'm getting a lot of warnings and then build fails: https://github.com/LucyGwilliamAdmin/nepal-data/runs/2804655196?check_suite_focus=true

[brockfanning]

Ok cool, I'll check it out now.

[brockfanning]

@LucyGwilliamAdmin Those issues were more about the Word input, but I think they are important issues so I threw fixes into this branch. Can you give it another try?

[LucyGwilliamAdmin]

@brockfanning not failing anymore but I'm not getting an SDMX folder in data build: https://lucygwilliamadmin.github.io/nepal-data/

[brockfanning]

@LucyGwilliamAdmin I think you may need to specify the DSD, like:

sdmx_output:
  dsd: CBS_DSD.xml
  constrain_meta: true

[brockfanning]

@LucyGwilliamAdmin Actually that works, but the metadata doesn't seem to be outputting right. Let me investigate a bit.

[brockfanning]

@LucyGwilliamAdmin I see the issue. It can't produce the metadata in SDMX unless it knows the ref area and reporting type. So try this config:

sdmx_output:
  dsd: CBS_DSD.xml
  constrain_meta: true
  meta_reporting_type: N
  meta_ref_area: NP

[LucyGwilliamAdmin]

@brockfanning I'm getting metadata xml output now - is there a way to validate?

[brockfanning]

@LucyGwilliamAdmin Not that I know of - given that the MSD has not been officially released yet I think this is pretty cutting-edge.

[LucyGwilliamAdmin]

Ok @brockfanning, since it's outputting are you happy for it to be merged at this point?

[brockfanning]

@LucyGwilliamAdmin Yep, I think it's good enough to merge at this point.