Performance issues related to `wrapWithDirectory: false`

[dvargas92495]

Hey all 👋 ,

I'm looking to store just blobs in IPFS and decided to give Web3.Storage a spin. It seems like the best way to do this for now is to use wrapWithDirectory: false on the put object. But when I do so, I'm noticing incredibly poor performance on the first download of the CID.

Basic flow:

const cid = await client.put(makeFileObjects(), { wrapWithDirectory: false }); // takes 1-2s
const response1 = await client.get(cid); // takes anywhere between 5-20s
const response2 = await client.get(cid); // <500ms
const response3 = await client.get(cid); // <500ms

I would expect that first response to be as performant as the second through nth responses. I've only noticed this happening when the wrapWithDirectory flag is set to false.

I created a minimally reproducible repo showcasing this behavior, based on the examples in the docs but using random content each time: https://github.com/dvargas92495/web3-storage-example

There's also a GitHub action running in the repo which runs the test. Latest run could be found here: https://github.com/dvargas92495/web3-storage-example/actions/runs/3299301527/jobs/5442509930

[dchoi27]

Thanks for opening the issue. Can you try re-running the test?

It doesn't really have to do with wrapWithDirectory. If the content is in a directory, when you fetch the content associated with the CID, you're not fetching the full data itself, but the list of files that the directory points to.

We did introduce some changes to the HTTP gateway that get fetches from to speed up reads. I just tested myself and it took a second or so - 5-20s isn't usual. But it's possible there could be gaps depending on things like where the test is run from?

[dvargas92495]

Still seeing poor performance both on GitHub actions and locally. Here's the last run action, where I ran it for 10 iterations and took an average: https://github.com/dvargas92495/web3-storage-example/actions/runs/3329847873/jobs/5511750758

Another issue I've been seeing with get: Sometimes, nondeterministically, a get call will fail with this error: Error: ENOENT: no such file or directory, open '/tmp/7349644161666794262006/bafkreica6botpkymdklldkdz3g25xzhcvaq37olgtvvyvay3rfhw2imw3a'

The latest run where I saw that error: https://github.com/dvargas92495/web3-storage-example/actions/runs/3329847873/jobs/5507583708

Note that there are no code changes between the two runs

[vasco-santos]

Hi @dvargas92495

It is common that follow up requests right away will be way faster given they will be stored within CDN LRU Cache.

I would not expect a gap between wrapWithDirectory and non wrapWithDirectory but will do some tests once I get a moment

The open error should be something with Github Actions? If fetch did not fail the file must have been fetched. You should check the status code of the response and see if that is 200 to guarantee request was successful .

[dchoi27]

@vasco-santos some of the read times for the first requests are kind of slow in their test results (though some of the speeds tested seem reasonable, and the average first read times seem skewed from a few data points). but in either case data this small shouldn’t ever take that long to read even the first time

[dvargas92495]

It is common that follow up requests right away will be way faster given they will be stored within CDN LRU Cache.

This makes sense, I'm mostly raising that the initial download is far slower than is practical, resorting to my application storing a copy on S3 for reads instead (in addition to IPFS)

The open error should be something with Github Actions?

I've seen the error locally and in my deployed Lambda as well. I'll add some additional monitoring around the status codes to investigate further

[vasco-santos]

Hi @dvargas92495

Apologies for taking longer to answer here. I did some tests and also looked through your testing code. I am seeing you are relying on client.get to grab the files, and this was probably the confusion with some of what we discussed before in this issue. Any reason for this?

Our JS client gets content from the network as a CAR file. This has benefits such as having the client to validate the response instead of trusting that the content served from the gateway is correct. However, we still have a gap on optimal read performance with this flow. Our reads pipeline (w3link) does not fully support reading CAR format. As a consequence of this, we are relying on dweb.link as the IPFS Gateway used from web3.storage client.get. In the meantime, if you switch from client.get to just do a HTTP request to {cid}.ipfs.w3s.link you should be able to see super fast reads right away.

We will be looking into supporting this in the entire dotstorage reads pipeline as part of https://github.com/nftstorage/nftstorage.link/issues/147

[dvargas92495]

Ah I was simply going through the docs for Retrieve and figured that since I was fetching from the node.js context, that using the client libraries would be more performant.

I'll try switching to {cid}.ipfs.w3s.link!

[vasco-santos]

@dvargas92495 let me know how it goes. Thanks for the pointer, we should look into recommending in the docs to use the gateway directly. That was written before we were running our own. Anyway, we should get to the point that client uses our own gateway pretty soon.

[dvargas92495]

Was able to confirm in my latest tests initial download speeds are now under 2s!

https://github.com/dvargas92495/web3-storage-example/actions/runs/3432163473/jobs/5721136346

This is much more usable for now, but still won't be able to replace S3 which is downloading in the 100ms range.

One thing to note - when wrapWithDirectory: false is set, the file is from the link directlry and not from the /filename.txt path. Might be something worth noting in the docs as I was getting 502s initially

[vasco-santos]

@dvargas92495 thanks for the tests. We are looking into setting up a system to monitor response times and work on improving those over time