Data Catalog for Bacalhau

[wdbaruni]

Summary: Implement a Data Catalog feature to allow compute nodes to publish metadata about the data they have or can access. This will enable users to submit jobs by defining the data they want to access, and the system will route the job to the most suitable nodes based on data access capabilities, proximity, and cost.

Description: In a distributed computing environment, efficiently managing and accessing data is crucial. This feature aims to create a Data Catalog that indexes metadata about available data across the Bacalhau network. The catalog will facilitate job submissions by enabling users to specify the data they need, and the system will automatically select the optimal nodes for job execution.

Key Features:

• Metadata Publication: Compute nodes can publish metadata about the data they hold or can access. Metadata should include:
  • Data type and format
  • Data size
  • Access permissions and requirements
  • Location and accessibility (e.g., public, private, restricted)
  • Any associated costs
• Indexing: The system will index this metadata to create a searchable catalog.
• Job Submission: Users can submit jobs by specifying the data they need to access. The system will:
  • Search the catalog for nodes that can access the specified data
  • Evaluate nodes based on proximity, access permissions, and associated costs
  • Route the job to the most suitable nodes
• Optimization: The system will optimize job routing to ensure efficient data access and cost-effective computation.

Benefits:

• Efficiency: Streamline job submissions by allowing users to focus on data requirements rather than node specifics.
• Cost-Effectiveness: Reduce costs by routing jobs to the most suitable and cost-effective nodes.
• Improved Access: Enhance data accessibility across the network by maintaining an up-to-date catalog.
• Scalability: Support scalable data management by distributing metadata publication and indexing.

Integration:

• Metadata API: Provide an API for compute nodes to publish and update metadata.
• Catalog Search API: Develop an API for requester nodes to search the data catalog for suitable nodes
• Routing Logic: Implement logic to evaluate nodes based on data access capabilities, proximity, and costs, and route jobs accordingly.

[wdbaruni]

For consideration https://github.com/bacalhau-project/bacalhau/issues/1010#issue-1434555585

[aronchick]

HOLY cow this would be great