Breakdown GDMR Metadata Catalogue epic

[billgeo]

Enabler

So that we understand what we need to do in upcoming increments, we want to fully understand the goals of, and break down, the GDMR epic for a metadata catalogue for all LINZ metadata

See section 4.3 https://confluence.linz.govt.nz/display/LIFFP/Geospatial+Data+Management+Strategic+Review+-+Final+Report

Create a Central Data Catalogue

Tooling and resources are required to enable the adoption and management of standardised metadata across LINZ. Providing metadata editing capability as part of a central data catalogue will allow strong metadata management practices.

The proposed data catalogue will keep a central index of metadata about all data inside of LINZ. This includes the structure, quality, definitions, lineage and usage of the data. By indexing all LINZ's metadata in a central catalogue, we can obtain data collection, management, and distribution metrics.

As the number of datasets in LINZ grows the catalogue will become a key enabler ensuring:

Productivity – Consumers of data can quickly find and get access to the data they need, while also identifying what additional data is available to inform their decision-making 

Agility – Consistency and collaboration across all LINZ’s datasets and their metadata 

Strong data foundations – A central location facilitates data governance 

Consistency – By using the same tool to manage all metadata, keeping information consistent becomes a natural side-effect. 

Discoverability – A central catalogue will enable the searching of descriptive information across all datasets 

Acceptance Criteria

• [x] We understand the epic and how to achieve the value
• [x] We understand the scope and have a prioritized list of user stories
• [x] We understand the overall system architecture and where the Geostore and Metadata Catalogue fit

Additional context

Tasks

• [x] Create a board/workspace to capture info/ideas
• [x] Develop system architecture with Geostore and Metadata Catalogue clearly shown
• [x] Run a workshop
• [x] Report/document results

Definition of Ready

• [ ] This story is ready to work on
  • [ ] Negotiable (team can decide how to design and implement)
  • [ ] Valuable (from a user perspective)
  • [ ] Estimate value applied (agreed by team)
  • [ ] Small (so as to fit within an iteration)
  • [ ] Testable (in principle, even if there isn't a test for it yet)
  • [ ] Environments are ready to meet definition of done
  • [ ] Resources required to implement will be ready
  • [ ] Everyone understands and agrees with the tasks to complete the story
  • [ ] Release value (e.g. Iteration 3) applied
  • [ ] Sprint value (e.g. Aug 1 - Aug 15) applied

Definition of Done

• [ ] This story is done:
  • [ ] Acceptance criteria completed
  • [ ] Automated tests are passing
  • [ ] Code is peer reviewed and pushed to master
  • [ ] Deployed successfully to test environment
  • [ ] Checked against CODING guidelines
  • [ ] Relevant new tasks are added to backlog and communicated to the team
  • [ ] Important decisions recorded in the issue ticket
  • [ ] Readme/Changelog/Diagrams are updated
  • [ ] Product Owner has approved acceptance criteria as complete
  • [ ] Meets non-functional requirements:
  • [ ] Scalability (data): Can scale to 300TB of data and 100,000,000 files and ability to increase 10% every year
  • [ ] Scability (users): Can scale to 100 concurrent users
  • [ ] Cost: Data can be stored at < 0.5 NZD per GB per year
  • [ ] Performance: A large dataset (500 GB and 50,000 files - e.g. Akl aerial imagery) can be validated, imported and stored within 24 hours
  • [ ] Accessibility: Can be used from LINZ networks and the public internet
  • [ ] Availability: System available 24 hours a day and 7 days a week, this does not include maintenance windows < 4 hours and does not include operational support
  • [ ] Recoverability: RPO of fully imported datasets < 4 hours, RTO of a single 3 TB dataset < 12 hours

[billgeo]

FYI @palmerj . Feel free to review what we've done so far and provide feedback.

Progress on this. We've created a user story map here: https://miro.com/welcomeonboard/b0k3aWZ0WGpiUExWQWMxUGVuelM3MndacGZlV1RHeUF3UFBoNEZqYmllamV2alV3OEhiWUMwSEtvQWpXaTdGbnwzMDc0NDU3MzQ3NjM5MDE3OTgw?invite_link_id=91963189844

With some high level users identified (needs to be tested).

@jacobus is working on the system design to make it clear where this fits, and who it's for. Somewhere in Excalidraw I think. Could you post a link here @jacobus ?

once we have strawmen for both of these I think we need two separate workshops to adjust and confirm

[palmerj]

Thanks @billgeo

Regarding the user stories it seems to me that the top 2 rows looks good, but I wonder about the breakdown int the detail beyond that right now without more user engagement etc. Maybe we could take some sort of interactive UI design process that looks at the end-to-end flow of the User Experience (UX) for those top level use cases and supporting actors? Maybe Jack can do that, he seems to have the skills...

Note from today's Hydro meeting I think it's clear there should be some early conversation with the people that run and use HITS. I'm keen to be involved in that.

[billgeo]

Halfway through

[billgeo]

Updated teh docs now. So I think this is finished. Will make any amendments as and when needed.