search

open-sdg / sdg-build

Python package to convert SDG-related data and metadata between formats
MIT License
5 stars 23 forks source link

SDMX and file helper functions #242

Closed brockfanning closed 3 years ago

brockfanning commented 3 years ago

This moves some commonly used file-related and SDMX-related code to dedicated files, where they can be re-used across classes. Some of the code in various classes is replaced with calls to these functions. Any class that needs to make remote calls has a new parameter called "request_params" which can be used to control the behavior of remote requests.

Six new helper functions are included -- but not used -- that may be useful in working with series codes in SDMX:

brockfanning commented 3 years ago

These helper functions don't seem to be working - I would hold off on any testing.

brockfanning commented 3 years ago

Ok, seems to be working now.

LucyGwilliamAdmin commented 3 years ago

@brockfanning can you give an example of what this might be used for so i know how to test please?

brockfanning commented 3 years ago

@LucyGwilliamAdmin This PR is mainly a refactoring to remove duplicate code, and also the addition of some utility functions. Those utility functions may not be particularly useful to most Open SDG platforms, but I can imagine using them to generate starting data in some cases. For example, maybe some raw source data is not broken down by indicator id but does have series, maybe "get_indicator_id_from_series_code" would be useful. Or maybe if you're trying to move towards SDMX, you might need to write an intermediary script to generate SERIES codes for each indicator, where "get_series_code_from_indicator_id" might be useful.

The one bit in this PR that is immediately useful in Open SDG is the "request_params" parameter that it adds to all fetching of remote files. This is needed in at least one known case (Cambodia's API seems to block certain user agents) and adds some flexibility for the future. For example, if a data source has some basic authentication user/password this could help support that.

LucyGwilliamAdmin commented 3 years ago

@brockfanning thanks for the info - so would they purely be used in sdg-build or also in country repos?

brockfanning commented 3 years ago

Right, those new utility functions are purely for use if/when needed. The use-case I have encountered so far (the reason for writing them) was to use sdg-build to convert some SDG data into SDMX - separate from Open SDG.

Otherwise this PR is mainly a refactoring to reduce code duplication, along with the one new "feature" of the request_params parameter to help with remote file fetching.