search

open-sdg / sdg-build

Python package to convert SDG-related data and metadata between formats
MIT License
6 stars 22 forks source link

Fixes for code and column mapping in SDMX output #288

Closed brockfanning closed 2 years ago

brockfanning commented 2 years ago
Q A
Feature branch/test site URL Link
Fixed issues Fixes #287
Related version 1.5.0-dev
Bugfix, feature or docs?
otis-bath commented 2 years ago

Hi @brockfanning, have this on a feature branch (http://uk-sdg-feature-branches.s3-website.eu-west-2.amazonaws.com/feature-column-and-code-mapping-fixes/) and the changes passed for the first time however the data still isn't being output at all here (http://uk-sdg-feature-branches.s3-website.eu-west-2.amazonaws.com/feature-column-and-code-mapping-fixes-data/sdmx-output.html).

It says that each time it is dropping the rows as the values are not supposedly allowed (https://github.com/ONSdigital/sdg-data/runs/4504927209?check_suite_focus=true#step:6:1070) - this is despite the column and code mappings being on the FB. Do you know what might be going wrong here? Thanks

brockfanning commented 2 years ago

@otis-bath Can you point me to a particular indicator that you are expecting would be SDMX compatible? I tried spot-checking one (3.7.1) but in that case I think it's correctly dropping all the rows, because it would not be SDMX compatible (for example all the rows have values in 'Contraceptive method' which is not an SDMX concept). But I suspect if you could narrow it down to 1 indicator that should work, then I'll be able to figure out locally why it's not working.

otis-bath commented 2 years ago

@brockfanning - Okay great, 1.1.1 or 2.a.1 should both be SDMX compatible, thanks

brockfanning commented 2 years ago

@otis-bath I've found when I run this locally, I do actually get SDMX output. I'm seeing output for these indicators: 1.1.1, 8.1.1, 8.2.1, 9.b.1, 17.1.2, 17.2.1, and 17.3.2.

I'm confused why it's not working in the feature branch. But in case there is a connection, here are a couple possibly-related issues:

  1. There is still that issue with the GeoJSON download failing, so I had to remove the map_layers config in order to test.
  2. Locally I noticed that my output was "Generic" instead of "StructureSpecific". This was because it was using an older version of the "sdmx" Python library. I've pushed some commits in this PR to enforce a minimum version of the sdmx library.

Maybe try again? Though you may need to remove map_layers if the GeoJSON issue hasn't been resolved.

otis-bath commented 2 years ago

Hi @brockfanning - thanks for having a look at this - now have the same 7 indicators working on the FB