search-futures

Key Repositories and Development Strands

Indexing

• Asset Scanner - Asset scanning framework. Provides the input/output plugins, item-description interface and holds shared utilities.
• Asset Generator - Converts a stream of URIs (filepaths) into assets. Outputs Asset dictionary and tuple(<item_id>,<filepath>)
• Item Generator - Summarises Assets to form Items. Can also run its own processors to enrich the item content. Input is a stream of item_ids and filepaths. Outputs Item and tuple(<collection_id>,<filepath>)
• Collection Generator - Summarises Items to form Collections. Can also run its own processors to enrich the Collection content. Input is a stream of collection_ids and filepaths. Outputs Collections
• Item Descriptions - Configuration files for the indexing workflow. They describe how to construct the assets, items and collections.
• Item Descriptions Deployment - Descriptions are also pushed here to build a data container for the indexing process.
• Asset Scanner Example - Example repo which launches in Binder to demonstrate the indexing workflow.
• STAC Indexing Deployment - Deployment of the STAC indexers in Kubernetes.

API

• Server Implementation - Our Elasticsearch implementation of the STAC API
• Deployment - Deployment for the API server
• Extensions:
  • filter - Built in extension to provide faceted search and more advanced queries based on CQL2 Elasticsearch queries made possible through pygeofilter-elasticsearch.
  • free-text - OpenAPI Schema to describe the API extension. Adds q parameter to search.
  • free-text-ext - Extension classes to use with the STAC FastAPI framework
  • context-collections - OpenAPI Schema to describe the API extension. Adds collection IDs in the context response.
  • context-collections-ext - Extension classes to use with the STAC FastAPI framework.
  • asset-search - OpenAPI Schema to describe the API extension. Adds /assets/search endpoint.
  • asset-list - OpenAPI Schema to describe the API extension. Addes /collections/<collectionId>/items/<itemId>/assets
  • discoverable-facets - Not yet created
• Asset Spec - Draft specification for assets

UI

• Application - ReactJS application to serve as a demo UI for STAC
• Deployment - Deployment of STAC UI in kubernetes

Vocabularies

• Vocabulary Generator - Generates vocabulary and mapping to canonical CEDA vocab from different source vocabs.
• Vocabulary API - API server to allow search for vocabs
• [Deployment]()

Upstream Repos to keep an eye on

• stac-fastapi - The framework for the STAC API Server
• pygeofilter - Constructs abstract syntax trees (AST) from CQL queries.

Early UML Architecture Diagrams

NOTE: PlantUML was used to generate diagrams.

Search API Interfaces and Middleware

Media Crawler and Ingest

Facet Extraction