panamax-rs / panamax

Mirror rustup and crates.io repositories, for offline Rust and cargo usage.
Apache License 2.0
433 stars 45 forks source link

empty "crates" directory #24

Closed panzerballett closed 3 years ago

panzerballett commented 3 years ago

i just restarted a fresh sync (using the mirror.toml file i got from panamax init with minor changes [contact=, base_url=]) and only the crates.io-index directory was updated. the crates directory remained empty.

$ panamax --version
panamax 0.2.2

hmmm. weird. while i was re-checking that what i wrote here was accurate, i re-ran panamax sync . and now 4 crates were mirrored: libnotcurses-sys, parametrizer, schemafy_core, schemafy_lib.

is this all just due to maintenance on the original mirror?

this is probably an unrelated issue but i also noticed that rust-src is not mirrored. this seems to be needed for rustup-init to work.

thanks & best regards!

panzerballett commented 3 years ago

a few minutes and several re-runs of panamax and i am at 6 crates: banyan libnotcurses-sys parametrizer schemafy schemafy_core schemafy_lib...

panzerballett commented 3 years ago

started sync from scratch.... things seem to work now, sorry!

k3d3 commented 3 years ago

No worries! I am curious though (since I don't think this should be happening at all), did Panamax come back with any errors when this happened?

panzerballett commented 3 years ago

i re-ran everything from scratch. this happened:

$ panamax sync .
Syncing Rustup repositories...
[1/5] Syncing rustup-init files... ███████████ 80/80 [00:00:06]
[2/5] Syncing latest stable...     █████████ 596/596 [00:09:43]
[3/5] Syncing latest beta...       █████████ 596/596 [00:09:49]
[4/5] Syncing latest nightly...    █████████ 642/642 [00:10:01]
[5/5] Cleaning old files...        █████████████ 0/0 [00:00:00]
Syncing Rustup repositories complete!
Syncing Crates repositories...
[1/3] Fetching crates.io-index...  ██ 1959500/1959500 [00:02:17]
[2/3] Syncing crates files...      ██████████████ 0/0 [00:00:00]
[3/3] Syncing index and config... 
Updating crates.io-index config failed: GitError(Error { code: -1, klass: 4, message: "corrupted loose reference file: FETCH_HEAD" })
You will need to sync again to finish this download.
Syncing Crates repositories complete!
Sync complete.

so there is an error...

a $ git log in the crates.io-index directory yields:

commit 839a17b3aec75f3710d5ecf3dae663df02848972 (HEAD -> master, origin/master, origin/HEAD)
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 13:28:52 2021 +0000

    Updating crate `transvoxel#0.1.1`

commit 459f58306a988d281e8383f63c58c3fe4cdde529
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 13:28:30 2021 +0000

    Updating crate `libpacket#0.1.0`

commit 67959271d37fabf21ef6ea7dd9779640d7c20d3d
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 13:25:31 2021 +0000

    Updating crate `libpacket-derive#0.1.0`
...

re-starting worked fine - no error. but only a single crate was mirrored:

$ ls crates/
wasmcloud-fs

in the next run i got 2 crates:

$ ls crates/
pfm  wasmcloud-fs

should i re-open the issue?

panzerballett commented 3 years ago

synced again after a few minutes:

$ git log
commit 487b076b5781042e8e1012f325ddb746185f7fe2 (HEAD -> master)
Author: Panamax <panamax@panamax>
Date:   Tue May 18 15:58:22 2021 +0200

    Rewrite config.json

commit f00d6335fefeeada2d05a751815a67608d625513 (origin/master, origin/HEAD)
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 13:39:58 2021 +0000

    Updating crate `pfm#0.0.6`

commit 70d120d55698ca5030132ecc0cfcb8188709d68f
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 13:37:19 2021 +0000

    Updating crate `wasmcloud-fs#0.4.1`

commit 839a17b3aec75f3710d5ecf3dae663df02848972
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 13:28:52 2021 +0000

    Updating crate `transvoxel#0.1.1`

and

$ ls crates/
pfm  wasmcloud-fs
k3d3 commented 3 years ago

Yep, this is definitely worth reopening if you're able to reproduce this even after cleaning up.

Thanks for the error message! I'll try to figure out what's going on.

k3d3 commented 3 years ago

Oh, one other question - how are you installing panamax? Are you using cargo install panamax, or are you cloning from the master branch and compiling that way?

If the former, I'd recommend doing

git clone https://github.com/panamax-rs/panamax
cd panamax
cargo install --path .

Just to make sure it's running off master, and it's the latest version.

panzerballett commented 3 years ago

i installed panamax via cargo install panamax.

current state of affairs:

$ ls panamax/crates
cargo-mono     faster-chars-count        matchit  neon-build   neon-runtime  ogg-opus  pixel-widgets  routerify-multipart  simple_asn1  wasmcloud-fs
crates_io_api  hierarchical_pathfinding  neon     neon-macros  neon-sys      pfm       png_pong       seacan               swc_bundler
$ git log

commit 845558e49157a21788957fad9b45f57eb1337b8d (HEAD -> master)
Author: Panamax <panamax@panamax>
Date:   Tue May 18 17:38:35 2021 +0200

    Rewrite config.json

commit 77aa0b0bc9d16875f89cb8e301a8ceef5c2b4e10 (origin/master, origin/HEAD)
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 15:27:03 2021 +0000

    Updating crate `matchit#0.4.1`

commit 66dbb2cc241ee51e9e41e0b92f7f02ab840bf5b3
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 15:20:02 2021 +0000

    Updating crate `faster-chars-count#0.2.1`

commit 0b8a2b551907d9d2a57d4c612f4f48c9a079c8fa
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 15:09:49 2021 +0000

    Updating crate `simple_asn1#0.5.3`

commit 912b089116022b13335e6ee0dc44582eb73bbf87
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 15:04:21 2021 +0000

    Updating crate `neon#0.8.2-napi`

commit d8c58f58bf436d034849d165531dcaed604b89ed
Author: bors <bors@rust-lang.org>
Date:   Tue May 18 15:03:24 2021 +0000

    Updating crate `seacan#0.1.0`

i will try to clone & build panamax next week & get back to you. thanks so far!

k3d3 commented 3 years ago

I've made a fix! The reason you were only getting a few crates is because when the initial sync was occurring, it was basically ignoring all crates up to that point. So further syncs would only grab the newly-added crates.

k3d3 commented 3 years ago

@panzerballett I've pushed a fix and updated the crates.io package. Let me know if you still have this issue, and feel free to re-open this issue if you do!

panzerballett commented 3 years ago

dear keith

thanks a lot for the quick fix! i did a full re-sync and most things work as expected again!

what does not work is a fresh rust installation. i can download rust-init but when i run it http://pamanax.internal/dist/2021-05-10/rust-src-1.52.1.tar.xz is not found. the directory http://pamanax.internal/dist/2021-05-10/ does exist and is populated - but no rust-src-* files are in there.

k3d3 commented 3 years ago

That's very strange. I'll take a look at that very shortly to see what's going on. I've also made a new issue for it, #27.