Meeting notes - 2022-08-08

[agstephens]

Search Futures Review Meeting - 2022-08-08

Agenda

• [x] High-level review of issues
  • [x] Sort into priority
• [x] Note down strands of work and progress, plans for each
• [x] Create Roadmap for 2022 with meaningful realistic milestones
• [x] Separate out projects for different activities

Summary of recent activities (Spring-Summer 2022)

Development of vocabulary system

• data structure for handling vocabulary terms
• approach includes general vocabulary terms that are relevant across different datasets/fields
• generator for CMIP6 vocabularies:
  • from a YAML description
  • mapped to general CEDA terms
• generic description template for each project
• FastAPI vocabulary service:
  • query content for a given project
  • query specific facet values
  • query mappings between the project-specific terms and general terms
  • query a term for a long name or description

To-do:

• adaptor to automatically write YAML description from external vocabulary (e.g. from CMIP6_CVs on GitHub)
  • needed for dynamic vocabularies that we track automatically

Architecture re-design

Done:

• moved item descriptions into a 3-layer description:
  • collection
  • item
  • asset
• rename item description as collection description
• moved asset/item/collection generators into the stac-generator repository

Considering:

• change architecture to move away from queues to trigger scanning
  • maybe we can poll ES (or a DB) to find out what work needs to be done
  • would require re-think of worker nodes:
  • single controller polls to find out what needs doing
  • sends jobs to worker nodes

Prototype for ESGF Search

...