Support curated 'groups' and 'subgroups' of datasets

[billgeo]

User Story

So that I can curate my datasets into groups and subgroups, as a Data Maintainer, I want to import and maintain a curated grouping of multiple datasets in both S3 path and the metadata path

e.g. imagery/aerial/dataset with each survey dataset grouped under 'aerial' and then 'imagery', also described in the logical structure here: https://confluence.linz.govt.nz/display/LIFFP/STAC+Field+Mapping

Structure should support the following

Metadata path

Catalog (all LINZ) -> Catalog (a group of datasets e..g. imagery) -> Catalog (a sub-group of datasets e.g. aerial/historic) -> Collection (a dataset e.g. a regional survey in a particular year) -> Items (metadata about the asset) -> Assets (the asset) Catalog (all LINZ) -> Catalog (a group of datasets e.g. Topographic) -> Collection (a dataset e.g. rivers) -> Items (metadata about the asset) -> Assets (the asset)

S3 path

bucket/group/subgroup/dataset bucket/group/dataset

Not sure if we should support more than three levels or not. I guess if we can allow for that, it would give the system maximum flexibility, but currently that's not the typical use case?

Acceptance Criteria

• [ ] Given a group or subgroup doesn't exist, when I request a group or subgroup be set up, then the group or subgroup is created in both S3 and the metdata
• [ ] Given a group or subgroup exists, when I request a group or subgroup be set up with the same path, then I am informed that it already exists
• [ ] Given a group or subgroup exists, when I import my data and request that it is added to the group or subgroup, then it is added to the group or subgroup in both the S3 and metadata paths

Additional context

STAC collections can be part of other collections and catalogs (plural). We can probably make this an arbitrarily deep nesting of collections, to enable datasets navigable by any relevant categorisation (regions and sub-regions, vector/photo, bathy/land, etc).

Tasks

• [ ] ...
• [ ] ...

Definition of Ready

• [ ] This story is ready to work on, according to the team's definition

Definition of Done

• [ ] This story is done, according to the team's definition

[billgeo]

FYI @l0b0 @imincik, after speaking with Victor about this, we were thinking it might be good to test the use cases with Daniel/Jack etc. And maybe there's a non-hierarchical way to meet the user need for grouping datasets in this way?

Probably best to talk about this together and come to some common understanding/agreement on where to go with this.

[billgeo]

Discussion on Slack here: https://linz.slack.com/archives/C01U5RBS7NC

Summary of discussion:

• This idea/requirement came out of analysis by the Metadata Project Team (many of whom are data maintainers or data owners).
• It is designed to support 'cataloguing' and 'browsing' metadata, which was decided was needed for 'users that want to find data, but don't know what search terms to search for'
• After discussion it was clear that:
  • A concept like this probably needs to be added to the Data Lake later, but isn't necessarily needed now
  • It will also be needed in systems like the LINZ data Service, data.govt.nz and other data catalog systems that we want ot publish too (although none of these support STAC at all yet).
  • While this could be implemented in the physical S3 structure, it doesn't necessarily need to be, it could be an abstract layer like a STAC API that allow browsing in this way, without changing the underlying S3 structure. This is TBD.
  • It would be good to keep the Data Lake design flexible enough to implement this easily later.

[billgeo]

Closing this until needed.