Consideration of granular metadata management.

[kaveenr]

Crosspost of https://github.com/ConsumerDataStandardsAustralia/standards-maintenance/issues/37

Description

Currently, the Data Holder Metadata Management takes place using periodic polling and Metadata AdminAPI call from the CDR registry for immediate updates.

The concern is scalability, Hypothetically with a large number of ADRs on-boarded with the CDR registry, the DH will have to query the complete ADR statuses through the Registry API.

The proposal is to make the APIs more granular allowing query of individual ADRs for metadata updating.

Area Affected

• CDR Registry GET /{industry}/data-recipients/brands/software-products/status
• CDR Registry GET /{industry}/data-recipients/status
• CDS AdminAPI POST ​/admin​/register​/metadata

Change Proposed

• CDR Registry GET /{industry}/data-recipients/brands/software-products/status
  • Pagination (in the case of the number of software products are unsendable in one payload)
  • Allow retrieving the status of a single software product using the softwareProductId using a query parameter or path parameter.
• CDR Registry GET /{industry}/data-recipients/status
  • Pagination (in the case of the number of ADRs are unsendable in one payload)
  • Allow retrieving the status of a single software product using the dataRecipientId using
• CDS AdminAPI POST ​/admin​/register​/metadata
  • Send the softwareProductId or dataRecipientId allowing invalidation of a singular software product/ data recipient.

This proposal will allow DHs to do JIT(Just In Time) provisioning of metadata instead of periodically having to completely retrieve the whole metadata list from the CDR registry.

[CDR-Register-Stream]

Thank you for your proposal.

For now the expectation is that the status APIs growth will not be insurmountable and given the repetitive nature of the content, will gzip nicely. ETags is also being considered to reduce the volume of data transferred.

Granularity of the remaining APIs may be considered in future iterations but will not be considered for the beginning of the regime

[kaveenr]

Thanks,

Just making sure the emphasis is on, as said on the original posting.

This proposal will allow DHs to do JIT(Just In Time) provisioning of metadata instead of periodically having to completely retrieve the whole metadata list from the CDR registry.

That will allow getting ADR statuses on a need to know basis, instead of keeping all ADR data by the DH.

cheers, Kaveen Rodrigo

[CDR-Register-Stream]

@kaveenr your emphasis is definitely noted. We will keep this ticket in place for future considerations of optimisation