implement replacement for dagcargo as it is deprecating

[alanshaw]

• [x] convert NFT storage infra to use w3up client to act as a normal customer of w3up and go through normal flow to add the data into w3up. Then all existing filecoin pipeline stuff just happens by default
  • [x] plan: https://github.com/nftstorage/nft.storage/pull/2518
  • [x] implement: https://github.com/nftstorage/nft.storage/pull/2522
• [x] uploads to nft.storage that go through w3up should eventually have 'Storage Providers' listed in the Files UI at https://nft.storage/files/ once w3up/w3-filecoin makes the deal
• [x] NFT storage read and filecoin info API's need to work against both w3up and legacy systems (and know which to use in each case)
  • [x] Get dump from riba and spin up a DB operated by us (Note that Riba needs to know exact query pattern to the FDW in order to export the proper subset of data (the full database is a completely unmanageable 6TiB monster))
  • [x] Ensure queries for deal info from old web3.storage API are routed to new database
• [x] @travis - deal information for w3up uploads should be retrieved through the w3up API

See also

• https://github.com/w3s-project/project-tracking/issues/23

Reviewer: @vasco-santos

[heyjay44]

Deals that were made pre Spade were dealt with and we do not need to worry about renewals.

[gobengo]

convert NFT storage infra to use w3up client to act as a normal customer of w3up and go through normal flow to add the data into w3up. Then all existing filecoin pipeline stuff just happens by default

This PR has an implementation: https://github.com/nftstorage/nft.storage/pull/2522

[gobengo]

w3up-client integration is deployed behind a feature switch in staging.nft.storage.

Good news is that it can be used to forward an 8MB upload to w3up without error.

Bad news is that it doesn't work in all circumstances, and we're still debugging why In testing we see that uploads that use it can fail in at least two ways:

• when the nft.storage client shards files into CARs before sending into nft.storage HTTP APIs, the HTTP request uploading the CAR can respond with a 503 and we see an error like "Error: Worker exceeded memory limit."
  • described in slack
• when the nft.storage client doesn't shard, but files are around 45M, the upload can fail due to a different uncaught exception from inside uploadCAR
  • described in slack

Update: This helped with the OOM errors and maybe more https://github.com/nftstorage/nft.storage/pull/2551

[alanshaw]

I am pg_dumping dagcargo tables: aggregate_entries, aggregates, deals, dags, metrics, metrics_log.

I will import into a new postgres in Heroku and check it can be attached as a foreign data wrapper to classic.nft.storage/old.web3.storage staging DBs. If successful I'll attach it to production DBs.

[alanshaw]

New DB in Heroku created: dagcargo ✅

Contains exported subset of dagcargo: aggregate_entries, aggregates, deals, dags, metrics, metrics_log.

Created 2 read only users nftstorage and web3storage - i.e. different user for each product but staging and production use the same user.

[hannahhoward]

@travis just testing api -- deploy to staging today, then prod soon, then we're done! 4/15/24