Spark v2: DDO, Allocators, Data Quality Improvements

[bajtos]

ETA: 2024-11-05

TL;DR: Each miner storing publicly retrievable data should show on the Spark public dashboard.

Spark is checking whether Storage Providers are offering public data retrievals for deals that are expected to be retrievable. To do so, we need to understand which deals are expected to be retriable. Earlier this year, the introduction of Direct Data Onboarding (DDO) and FIL+ Allocator Pathways significantly changed the Filecoin landscape, and our previous solution is no longer sufficient. The goal of this milestone is to improve Spark to support DDO & Allocators.

The second objective is to improve quality of data produced by Spark and filter out more misbehaving nodes.

Resources:

• https://www.notion.so/spacemeridian/Spark-v2-40138e6879a04e8589925b3ee9ae11b1
• https://docs.google.com/document/d/1YvFYYUwP6VibKyTeHLd89mrkB1S5sVjq9ydFCGbBfrw/edit

children:

• Support New Allocators:
• https://github.com/filecoin-station/fil-deal-ingester/pull/16
• Data Quality Improvements:
• https://github.com/filecoin-station/spark/issues/46
• https://github.com/filecoin-station/spark/issues/43
• https://github.com/filecoin-station/spark/issues/27
• https://github.com/filecoin-station/spark/issues/38
• https://github.com/space-meridian/roadmap/issues/59
• https://github.com/space-meridian/roadmap/issues/156
• DDO deals:
• https://github.com/space-meridian/roadmap/issues/128
• https://github.com/space-meridian/roadmap/issues/143
• https://github.com/space-meridian/roadmap/issues/144
• https://github.com/space-meridian/roadmap/issues/147
• Tooling & Data Access:
• https://github.com/filecoin-station/spark/issues/78
• https://github.com/space-meridian/roadmap/issues/139
• Incentives:
• https://github.com/space-meridian/roadmap/issues/142
• https://github.com/space-meridian/roadmap/issues/146

[MikeH1999]

Sorry, I'm wondering how long this will take.

[juliangruber]

It's currently planned to land this year https://github.com/space-meridian/roadmap/issues/113