search

panamax-rs / panamax

Mirror rustup and crates.io repositories, for offline Rust and cargo usage.
Apache License 2.0
427 stars 45 forks source link

[BUG] inconsistent state may occur #125

Closed jxlpzqc closed 7 months ago

jxlpzqc commented 7 months ago

Hi, we deploy panamax on our mirror site, and find a crate package exists in crates-index, but misses in crates files.

https://github.com/panamax-rs/panamax/blob/2024b5bab322724da7b6281d8d7dc62e33a6b146/src/crates.rs#L295-L311

In above code block, the program ignore DownloadError and fast-forward the git repo.

We suppose downloading of one crate meets network error. Next time, this failed crate is absolutely not in diff result of origin/master and master because fast-forward has been done, which leads to this version of crates will never get downloaded since this time.

Even worse, when syncing logs are cleared periodically, it is not easy to find which states have been broken. (the situation we meet.)

We suggest when error occurs, just stop going on, and wait for syncing next time to avoid inconsistent state occurs.

rust1248 commented 7 months ago

Couldn't you use panamax verify?

jxlpzqc commented 7 months ago

Sure, panamax verify may address this issue to some extent, but when deployed in an automatic situation, how often should verifing runs?

Anyhow, the inconsistent states has been entered when download failed. Why not just cancel this broken job?

k3d3 commented 7 months ago

The reason Panamax works the way it currently does is because when I first made the program, there were several crates which would repeatedly return a 403 error, and would break every single attempt to mirror the repository. That's why it logs rather than fails, because originally it would have failed every time.

@rust1248 is right; this is the use case for the verify command. The idea is that depending on how averse you are to having an incomplete mirror, you can either run verify after each sync, or on a regular interval like once a week, or at its lowest, only when a mirror user complains that a package is missing.

With that said, I do feel like the logging story for Panamax is lacking, and doesn't lend the best interface for automation right now. I need to work on that.

jxlpzqc commented 7 months ago

@k3d3 Thank you very much for giving such a detailed answer. We are also so grateful to panamax due to its huge role in our mirror site. I have fully understood the reason for this design, and will close this issue.