Add support for partial references

[sosna]

The SDMX RESTful API currently features a detail parameter. When references are requested for a structural metadata query, all available information for these references will be returned, unless the value referencestubs is used.

SDMX item schemes (codelists, concept schemes, category schemes, etc.) feature a isPartial attribute, but this functionality is currently not used in the RESTful API. It would be nice to add a referencepartial (or something along these lines) to the detail parameter, so that only items relevant in a certain context get returned.

Let's take for example a query for a DSD and its children (i.e. codelists and concept schemes). When using full (the default), all concepts contained in the concept schemes will be returned, including those not used in the DSD. When using referencestubs, no concept will be returned. If we add referencepartial, we could retrieve the concepts used in the DSD and only those, i.e. only the ones relevant to the context of the query. It would be the same for other item schemes, such as when retrieving the codelists referenced by hierarchical codelists or content constraints.

[dosse]

I would strongly agree to such a feature, and this feature becomes now urgent for SIS-CC. Today, codelists can be huge and impractical in online user interfaces, and they are likely to be restricted with valid content constraints to small portions for usage in a DSD, dataflow or provision agreement. Most frequently, the client only needs the relevant codes of codelists used in DSD, dataflows and provision agreements. It would even be sensible to define partial codelists as default when retrieving them as references in queries for DSDs, dataflows, and provision agreements.

[egreising]

I also strongly support this new option for details. The verbosity of the responses including so many useless artifacts and or items results detrimental for the client.

[chris-beer]

+1, FWIW (given I'm not formally a TWG rep atm).

If combined with the HEAD method, I think this could create quite powerful and simple interrogation tools if clients could see the size of the full code list in advance and then know to request a partial list (effectively allowing a WHERE constraint on a query).

Combined with #50, I think these approaches would also begin to provide good mitigation against inadvertent calls for whole cubes which can be very large and complex.

Cheers

Chris

Chris Beer Australian Bureau of Statistics +61 2 6252 5614   chris.beer@abs.gov.au     www.abs.gov.au

The Australian Bureau of Statistics acknowledges the traditional custodians of country throughout Australia and recognises their continuing connection to land, waters and community. We pay our respects to them and their cultures, and elders, both past and present

From: Jens Dossé notifications@github.com To: sdmx-twg/sdmx-rest sdmx-rest@noreply.github.com, Cc: Subscribed subscribed@noreply.github.com Date: 21/10/2017 05:10 AM Subject: Re: [sdmx-twg/sdmx-rest] Add support for partial references (#45)

I would strongly agree to such a feature, and this feature becomes now urgent for SIS-CC. Today, codelists can be huge and impractical in online user interfaces, and they are likely to be restricted with valid content constraints to small portions for usage in a DSD, dataflow or provision agreement. Most frequently, the client only needs the relevant codes of codelists used in DSD, dataflows and provision agreements. It would even be sensible to define partial codelists as default when retrieving them as references in queries for DSDs, dataflows, and provision agreements.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[achronis]

Hi,

To avoid any confusion maybe the new SDMX documentation for referencepartial could also mention the use of content constraints linked to dataflow or DSD to limit an item scheme.

Or possibly have a table with one column the requested resource and the other the items that can be constrained with referencepartial similar to the table at Applicability and meaning of references attribute

e.g.

Maintainable artefact	Limited artefact	Limited artefacts using content constraints
dataflow	conceptscheme	codelist

Else I fear the current documentation could be interpreted as only direct references should be limited with referencepartial i.e. using reference partial in a datastructure request will constrain only concept schemes.

[sosna]

Thanks @achronis. We'll make sure to update the wiki with this information when the next version of the API gets released later on this year.

[sosna]

Addressed in the upcoming v1.3.0 release