Create input plugin for ESGF Solr CMIP(5/6) data to generate STAC content

[rsmith013]

The ESGF Solr index has many metadata fields which are important for ESGF.

In order for STAC to be seen as a viable alternative, we need to show the community the future and what it looks like.

[agstephens]

The suggestion is:

• [x] Understand the basic search capabilities of ESGF (dataset and file level search; querying available facets for search): https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html
• [x] Discuss with Alan how to query/pull contents efficiently from the CEDA ESGF Index Node
• [x] Create the appropriate item-descriptions for building the records in our STAC index
• [x] Build an input plugin for ESGF CMIP6 and CMIP5 records. (NOTE: there is lots of non-CMIP data on ESGF, let's ignore that for now)
• [x] Build a media handler for esgf-solr
  • [ ] item handler
  • [x] asset handler
• [x] Create a new "esgf-cmip" index to house this data
  • Index names:
  • esgf-solr-assets-2022-03-10
  • esgf-solr-items-2022-03-10
  • esgf-solr-collections-2022-03-10
• [x] Run the pipeline to load the records into STAC/Elasticsearch

The difference in converting from Solr to STAC items vs converting to Elasticsearch is the structure. Due to the different field types required for geo-spatial and temporal queries, the JSON structure differs slightly.

[Mahir-Sparkess]

• Asset-generator - media handler: https://github.com/cedadev/asset-generator/pull/25
• Asset-scanner - input plugin: https://github.com/cedadev/asset-scanner/pull/35
• Collection-generator: https://github.com/cedadev/collection-generator/pull/7
• Item-generator: https://github.com/cedadev/item-generator/pull/50
• item-description: cmip6 deploy description and description repository

Leaving a comment here to consolidate everything into one place.