Move from Filecoin State Market deals to Filecoin Deals

[davidgasquez]

The Portal is only ingesting State Market Deals (deals notifying the f05 actor). In the last few months, there are other kinds of deals that don't appear in the sources I'm using (State Market Deals).

Need to figure out a way to ingest all the deals and surface which addresses are notifying (or not) and in which sectors they appear.

[davidgasquez]

There are two potential sources for deals data.

1. Read and parse all the successful messages with ProveCommitSectors3 and ProveReplicaUpdates3 methods.
2. Look at the events of built-in actors and parse the different deal related ones.

The built-in actors route seems easier, but I'm not sure events are 100% reliable. Going the message's way might be more accurate, but is harder and needs the business logic to be replicated downstream.

[davidgasquez]

BigQuery SQL to get the data from ProveCommitSectors3.

  SELECT 
    height, 
    cid,
    `from` AS from_address,
    `to` AS to_address,
    value,
    method,
    params,
    JSON_EXTRACT(params, "$.SectorProofs") AS sector_proofs,
    JSON_EXTRACT(params, "$.AggregateProof") AS sector_aggregate_proof,
    JSON_EXTRACT(params, "$.SectorActivations") AS sector_activations,
    JSON_EXTRACT(params, "$.AggregateProofType") AS sector_aggregate_proof_type,
    JSON_EXTRACT(params, "$.RequireActivationSuccess") AS sector_require_activation_success,
    JSON_EXTRACT(params, "$.RequireNotificationSuccess") AS sector_require_notification_success,
  FROM `lily-data.lily.parsed_messages`
  WHERE method = 'ProveCommitSectors3'

[davidgasquez]

The folks at Meridian will work on this (https://github.com/space-meridian/roadmap/issues/115). Will be great to reach out and learn more about their approach!

[davidgasquez]

We should get at least all verified f06 deals, which should include DDO deals / data.

[davidgasquez]

We can get the DDO onboarded data from Spacescope APIs. Sample URL:

https://api.spacescope.io/v2/deals/direct_data_onboarding?start_date=2024-04-26&end_date=2024-05-02