search

web3-storage / web3.storage

DEPRECATED ⁂ The simple file storage service for IPFS & Filecoin
https://web3.storage
Other
503 stars 119 forks source link

Ability to upload (split) CAR files using webUI #1621

Open hsn10 opened 2 years ago

hsn10 commented 2 years ago

Because most times you need to split CAR file into smaller parts (100MB) to get around upload limit, some way of signalling if all chunks from root CID got uploaded is needed in CAR upload form.

Probably good to add some signalling to JS API too. You upload bunch of CAR files using JS API and get error - few chunks are missing.

vasco-santos commented 2 years ago

This needs prioritization and design work to create a plan on how to split cars without exploding browser memory.

Today, we recommend using JS Client or w3 CLI for this kind of use cases.

vasco-santos commented 2 years ago

Probably good to add some signalling to JS API too. You upload bunch of CAR files using JS API and get error - few chunks are missing.

Can you give us more information on this bit @hsn10 ?

hsn10 commented 2 years ago

In CAR file there is one or more root nodes. Probably only one root is supported by most tools.

There needs to be API for:

  1. if I give it local CAR file it will return root CID.
  2. query status of uploaded root CID. it will tell me if everything linked from this root is correctly uploaded using split CAR parts. if everything is uploaded then IPFS will pin root on DAG IMPORT. So if CID is pinned then return true.
  3. server will need to keep CAR file with root around and after each CAR part upload it will need to retry to dag import root CAR to check if all parts are already uploaded. If root gets pinned, all parts are there.
  4. I assume that CAR parts holding additional data which will link to CAR file with root will have zero root, so we can detect if uploaded CAR file is supposed to be the main one or its additional data.