panamax-rs / panamax

Mirror rustup and crates.io repositories, for offline Rust and cargo usage.
Apache License 2.0
427 stars 45 forks source link

[Question] Can I start sync while the panamax is serving? #122

Closed windoze closed 1 month ago

windoze commented 8 months ago

Is it safe to sync and serve at the same time?

I have a mirror fully-synced, but I'd like to update the mirror without stopping serving, is this a safe action?

Thanks, Chen

windoze commented 8 months ago

It's pretty hard to produce a real error, but I roughly looked at the code, and I found:

  1. Panamax just serves static files, so new additions won't affect anything existing.
  2. Panamax syncs in 2 stages, index first and then crates, the index updating is merely a git pull and ff checkout, it should be safe in almost every case.
  3. The race condition exists after index updating and before all crates are downloaded. When new crates are being downloaded, it's possible that the client gets 404 or even half-downloaded files because the index already got updated hence the client assumes the crate exists on the server side, but I guess re-run cargo can resolve most problems.

Given the situation I guess the fix won't be very straightforward, but otho the issue is not very significant either.

I'm thinking of combining the serving and syncing functions into one process, so they can talk to each other and lock/pause the other one when needed, it could be the simplest way.

k3d3 commented 1 month ago

Note that while the index update is a git pull and checkout, it only updates the master branch after the crates have been downloaded. This is because it uses the difference between origin/master and master to figure out which new crates to download.

If there are any race conditions beyond that, they're probably not very significant. At least, nothing that would cause corruption, or anything more than just redownloading dependencies a second time.

So to answer your question, yes, it should be safe to sync and serve at the same time - in fact, panamax is built with that use case in mind.