search

livepeer / task-runner

Background service that executes tasks from the Livepeer API. Mainly used for VOD.
MIT License
3 stars 2 forks source link

Support ipfs:// and ar:// URLs in Catalyst upload task #79

Closed yondonfu closed 1 year ago

yondonfu commented 2 years ago

https://github.com/livepeer/task-runner/pull/76 and https://github.com/livepeer/task-runner/pull/77 added support for ipfs:// and ar:// URLs for the import task which is used for the current VOD pipeline. We'll want to support these URLs for the upload task used for the new Catalyst VOD pipeline as well.

Since Catalyst will receive a URL from task-runner, a few options could be:

0xcadams commented 2 years ago

Leaving a comment here since this functionality is moving to Catalyst - currently we are seeing a lot of 504 Gateway Time-out on the first attempt to fetch the file from a list of gateways. On a second attempt, these are successful, since the gateway has time to find the file on the IPFS network.

We should add a retry/backoff mechanism to make this more resilient to a slow IPFS network.

yondonfu commented 2 years ago

Jotting down some notes for further thought:

An inefficiency with the current approach is that even if a user knows that a CID is cached at an IPFS gateway URL (i.e. because it was pinned with that gateway) we could end up trying to fetch the CID from a gateway that does not have it cached. For example, if a CID was first pinned at a Pinata gateway URL, we might try fetching the CID from w3link which would need to pull the CID from the Pinata gateway. In the best case, the gateway that pins the CID is the gateway that the CID is fetched from (i.e. data was uploaded to w3s and is fetched through w3link). In the worst case, the gateway that the CID is fetched from needs to wait to pull data from one of its peers (or even worse it needs to discover the peer using the public DHT).

We should be able to improve on this a bit by first trying to directly download the CAR file from the gateway (if a gateway URL is provided) - we can verify the CAR file to ensure that the data matches the requested CID. In the scenario where the gateway does not support CAR file responses (perhaps because it is an old implementation), we can fallback to the current approach of using a list of trusted public gateways to fetch the CID from. In the best case, we would download the CAR file directly from the gateway that pinned the CID. In the worst case, we fallback to the current approach.

yondonfu commented 1 year ago

This is already implemented here.