search

tech-greedy / singularity

New node software for large-scale clients with PB-scale data onboarding to Filecoin network
Other
39 stars 18 forks source link

Native estuary integration #93

Open anjor opened 2 years ago

anjor commented 2 years ago

// Chatted about this offline with @xinaxu and @kernelogic

We could consider a native integration with estuary. There are two workflows I see here:

  1. Use singularity for deal preparation, and then upload the generated car files to the hosted estuary node at https://estuary.tech/
  2. Use singularity for deal preparation, and then onboard data using an estuary node that's being run by the SP (or whoever is onboarding the data).

For (2) there might be some dev work required on the estuary side to be able to ingest data that is locally hosted -- I will need to double check this.

I would expect the singularity config file to have the standard estuary config (URL, API) and then an additional option in the deal making service to upload to estuary.

jimmylee commented 2 years ago

This is awesome, I love the idea of using singularity for deal preparation and sending that data to our hosted Estuary node or any other self hosted Estuary node.