Converts a file into a CAR file and sends it to web3.storage (the file is already uploaded and has its own CID, but it has no notion of any directory structure).
Stores the file CID in the in-memory DAG structure
The DAG structure is identified by a unique string pubId
Calling Publish() uploads the whole dag directory structure into web3.storage and therefore makes all the HLS manifest files usable.
Additional comments
W3S Driver is stateful (because it needs to store the DAG structure); note however that only file CIDs are stored in memory, the actual file data are not stored anywhere
3 external process command calls are used:
ipfs-car --pack - it archives any file into a CAR file, I think that technically we could do the same using Golang and go-car library, but I think it may be more work and it may be more error-prone, since big files are actually sharded in the IPFS structures
w3 can store add - it uploads a CAR into web3.storage
w3 can upload add - it binds together a number of CARs and publishes their root with the public URL
The authentication and authorization will be handled in a separate PR (https://github.com/livepeer/catalyst/issues/383), so don't worry that key and proof variables are not used right now; in this PR the local w3 credentials are used.
During the Publish() step, the each subdirectory is uploaded as a separate CAR; technically, we should not need it, we should just create one CAR with the whole subdirectories and upload it. However, currently the go-car library fails for these scenarios, so that's the workaround. Here's the go-car PR which fixes the issue: https://github.com/ipld/go-car/pull/357 UPDATE: go-car merged my PR, so we there is no longer need to create each subdirectory separately. Updated the PR.
Add web3.storage driver.
fix https://github.com/livepeer/catalyst/issues/375
How does it work?
SaveData()does the following:pubIdPublish()uploads the whole dag directory structure into web3.storage and therefore makes all the HLS manifest files usable.Additional comments
ipfs-car --pack- it archives any file into a CAR file, I think that technically we could do the same using Golang andgo-carlibrary, but I think it may be more work and it may be more error-prone, since big files are actually sharded in the IPFS structuresw3 can store add- it uploads a CAR into web3.storagew3 can upload add- it binds together a number of CARs and publishes their root with the public URLkeyandproofvariables are not used right now; in this PR the localw3credentials are used.Publish()step,the each subdirectory is uploaded as a separate CAR; technically, we should not need it, we should just create one CAR with the whole subdirectories and upload it. However, currently theUPDATE:go-carlibrary fails for these scenarios, so that's the workaround. Here's the go-car PR which fixes the issue: https://github.com/ipld/go-car/pull/357go-carmerged my PR, so we there is no longer need to create each subdirectory separately. Updated the PR.