search

sdmx-twg / sdmx-json

This repository is used for maintaining the SDMX-JSON message specifications.
54 stars 20 forks source link

Browse by Topic promotes out of date workflow #86

Closed agent96 closed 2 years ago

agent96 commented 4 years ago

browse-by-topic.md Query 2: Find data in the selected dataflow, using concept filters

This step promotes using the structure query for dataflow and referencing constraints to build up a picture of what data is available. This mechanism is out of date and should not be the promoted mechanism to support this use case. The mechanism to support his use case is via the Data Availability API which is part of the SDMX REST query specification and documented in following places:

https://github.com/sdmx-twg/sdmx-rest/blob/master/v2_1/ws/rest/docs/4_6_1_other_queries.md https://github.com/sdmx-twg/sdmx-rest/wiki

The data availability mechanism was put in place to solve a number of issues with the currently documented process. This is mainly that the REST API for structures does not distinguish between a constraint which is used to restrict data reported vs data that exists – therefore the response to the currenlty documented process may contain much more information then what is required by the client, leaving it up to the client to pick through the response to interpret it. In addition as the structure API tends to return fixed structures, which are maintained by a user, it was deemed better to have a separate API which could return dynamically created structural metadata based on the data that exists, i.e the constraint does not really exist and is not physically maintained, it is created based on what data exists – this allows for dynamic behaviour, such as asking the question ‘what Countries have data for this dataflow’ or ‘If I select this Frequency, what Indicators are valid’.

Therefore, this document should be modified to promote the API which was created specifically to solve this use case.

sosna commented 4 years ago

I fully agree with @agent96! Indeed, the availableconstraint resource was added to the REST API in 2018 to support the type of use cases described in browse-by-topic.md, so it would be good if that document could reflect that.

dosse commented 4 years ago

Yes, thanks for highlighting this. Unfortunately, this page was overseen when releasing the structure formats since data availability was elaborated in parallel. Since you were driving forward the data availability topic, maybe you would like preparing a pull request to enhance this "requirements" page? Alternatively, the page could simply be removed, maybe it's not that relevant for the specialities of the SDMX-JSON structure format and more appropriate in the wiki anyway?