search

leszko / dstorage-testing

0 stars 0 forks source link

w3s.link gateway improvements #1

Open olizilla opened 1 year ago

olizilla commented 1 year ago

We recently made the w3s.link gateway go much faster for content uploaded to web3.storage, with the introduction of the freeway.

Your test-case helps identify an issue which we intend to fix! I tried out your test video with an online m3u8 player, and playback is fast, and from the headers I can see all the response are coming from freeway. At 1080p on 2x playback speed I'm able to playback the entire video without waiting. Milage may vary depending on network speed, but here is where i tested it. Let me know if you get better performance with it:

https://www.m3u8player.live/#type=m3u8&src=https%3A%2F%2Fbafybeibowt4rc4g2lahrl4e7esuwykfrf35ciid6i6lzbcz33czk3wzcjq.ipfs.w3s.link%2Fcracow_video%2Findex.m3u8

We suspect that ffplay may be sending range requests even though the video is chunked already. We can't yet serve range-requests via freeway... so if that's the case the video would be served via the slower path. We'll post updates on progress on the fix here https://github.com/web3-storage/gateway-lib/issues/12

leszko commented 1 year ago

@olizilla Thanks for the quick comment!

I double check and right, the read performance was related to ffplay sending range requests. I checked with a few online players and the performance was better.

So, to summarize the current state of art:

  1. read performance - it's good, the only issue is that web3.storage does not support range requests
  2. write performance - it takes quite a while (5-30 min) until files are available at the expected speed and this is something the web3.storage team plans to tackle in Q1

Is that correct? @olizilla

olizilla commented 1 year ago

read performance - it's good, the only issue is that web3.storage does not support range requests

Range requests are supported but they are handled by a slower code path today. We will fix this so that range requests are fast too! I can't say when, but Q1 is a very safe estimate.

write performance - it takes quite a while (5-30 min) until files are available at the expected speed

this should be fixed since we introduced freeway a few weeks ago! The slow time to availability you experienced is connected to the range-requests as well!

If you upload your files to using the web3.storage api client (or manually to api.web3.storage) then the current implementation will make your uploads available from the w3s.link gateway as soon as you get the response ...assuming you dont send range headers, and you sent a complete DAG, which is the case when uploading regular files rather than custom IPLD graphs. If you do request it with range headers today, then the slower code path is used which also is less predictable as to when the gateway will be able to provide it.

But again, this limitation will be fixed. You will be able to have instant range requests soon! if that isn't what you find in your testing (without range headers), please do let us know!

leszko commented 1 year ago

Thanks for the explanation @olizilla

I checked again without using any range requests. And right, the download speed is ok. Still I see a slight difference between the download "just after upload" and download "a few minutes later". I tried now 3 times, uploading a set of video files and then downloading them.

Video N Download time just after upload Download time later a few minutes later
1 29.583134s 21.553767834s
2 2m50.00302425s 1m5.003229834s
3 30.002897625s 15.000807834s

In any case, the video is actually possible to playback just after the upload.