Open juhp opened 3 months ago
LTS-22.22:
Tue May 14 05:38:42 PM UTC 2024
:
Uploading docs to S3
Creating hoogle/orig.tar
Shelling out to AWS CLI to upload docs
Uploading snapshot definition to Github
:
real 248m5.723s
user 0m0.781s
sys 0m0.218s
Tue May 14 09:46:48 PM UTC 2024
Nightly:
Wed May 15 01:18:19 AM UTC 2024
:
real 188m39.820s user 0m0.680s
sys 0m0.158s
Wed May 15 04:26:59 AM UTC 2024
Thanks for opening a ticket! From earlier discussion had been under the impression that this wasn't a blocker. I'll ping cloudflare to see if they've made any progress. It should not be happening.
Btw what's the actual evidence that the docs upload is the slow part of the process? I was unable to reproduce slow uploads, myself.
(That's not to say I don't believe it's slow. I'm just not sure how to reproduce it.)
It looks like the actual command run by curator upload-docs
is aws s3 cp --only-show-errors --recursive --acl public-read --cache-control maxage=31536000
.
I didn't try --recursive
-- maybe that has an effect.
EDIT: Wasn't fast enough to see the end. And log files already have been overwritten. But that probably doesn't change the basic observation. Upload seems to be limited by CF performance. I guess @chreekat is right - awscli seems to serialize file operations and doesn't use parallel requests (with this none-streaming approach even the normal latency would limit throughput)? I will try to save the log during the next lts.
Currently nightly is uploading while creating the debug-logs using the new curator version - but as it seems CF R2 is slow - upload speed around 1.3 MiB/s (and we upload several GB's). Will update this comment once the upload finishes...
Interesting. I wonder if the problem is a fixed cost per file. I tried uploading one big file and it was fast. But curator
is uploading a large number of small files. If there's a significant startup cost for each file, that could explain the issue.
That makes sense. From previous interactions with GCS, I observed a similar behavior. It's faster to archive all files into a single large upload than uploading individually.
Last log entry from Nightly-Upload - uploaded 9.8 GB by 1.0 MiB/s => ~ 2,75 h; one way to maybe improve upload time would be parallelization (e.g. there are tools that support parallel uploads - e.g. https://rclone.org/ and others). Maybe something you want to try @chreekat? 😉
(Presumably a single file upload is not an option, right?)
My guess was wrong - also awscli uses multiple threads (https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#max-concurrent-requests). So maybe we could tweak the setting to use e.g. 50 parallel connections?
@alaendle I just heard from CF and they have essentially recommended the same action. They recommended rclone as a better alternative since apparently it "gives more control over increasing concurrency/number of transfers".
I do wonder if there was some global aws config in the old system that implicitly increased the concurrency? I have a hard time believing that a sequential upload of all the files would be fast in any circumstance. Maybe I missed it when I was implementing the handover.
Sorry this issue has dragged on. I have less time now since my HF contract has been reduced to 20% time. But it's still on my list...
I am adding rclone to the nightly image now - though I dunno how to use it
With the switch from S3 to CF storage for stackage haddock documentation, uploads with the awscli to the CloudFlare via their S3 compatibility is really taking too long.
Nightly currently takes about 3 hours and LTS around 4 hours. In the past with S3 it just took minutes to do those uploads. The long delay basically prevents curators continuing to do work while the upload is happening.