Find common failures

[alsuren]

Since we're in a polars mood at the moment, I thought I would try using it to analyse our logs.

It's pretty aggressively cached (to keep the runtime around 0.3s when developing), so you currently have to manually rm -rf /tmp/get_runs if you want a fresh set of runs.

I recommend something like cronjob_scripts/find-common-failures.py > /tmp/fail.md && code /tmp/fail, then use vscode's preview mode to view the result.

Example output

| message | count | example_job | example_url | | --- | --- | --- | --- | | error: failed to run custom build command for `openssl-sys ` | 26 | rust-latest-1.4.1-armv7-unknown-linux-gnueabihf | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810332726 | | error: could not compile `winapi` (lib) due to 286 previous errors | 5 | project_init-3.1.23-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810137594 | | error: linking with `/home/runner/.cache/cargo-zigbuild/0.19.1/zigcc-armv7-unknown-linux-gnueabihf.2.17-96fe.sh` failed: exit status: 1 | 4 | cicada-0.9.38-armv7-unknown-linux-gnueabihf | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809506686 | | error: linking with `/home/runner/.cache/cargo-zigbuild/0.19.1/zigcc-x86_64-unknown-linux-musl-289f.sh` failed: exit status: 1 | 3 | wasm-opt-0.116.1-x86_64-unknown-linux-musl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809983297 | | error: could not compile `time` (lib) due to 1 previous error | 3 | llvm-cov-pretty-0.1.9-x86_64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809877790 | | error: failed to run custom build command for `ring ` | 3 | scryer-prolog-0.9.4-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810108049 | | error: `box_syntax` has been removed | 2 | rerast-0.1.96-aarch64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810160466 | | error: linking with `/home/runner/.cache/cargo-zigbuild/0.19.1/zigcc-armv7-unknown-linux-musleabihf-7c85.sh` failed: exit status: 1 | 2 | sane-fmt-0.18.1-armv7-unknown-linux-musleabihf | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810302002 | | error: there is nothing to install in `intel-mkl-tool v0.8.1`, because it has no binaries | 2 | intel-mkl-tool-0.8.1-armv7-unknown-linux-musleabihf | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810283384 | | error: could not compile `rerast` (lib) due to 64 previous errors | 2 | rerast-0.1.96-aarch64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810160466 | | error: could not compile `users` (lib) due to 41 previous errors | 2 | natls-2.1.14-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810122999 | | error[E0308]: mismatched types | 2 | cargo-kcov-0.5.2-x86_64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810029996 | | error[E0425]: cannot find function `real_main` in this scope | 2 | cargo-profiler-0.1.6-x86_64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810022705 | | warning: none of the package's binaries are available for install using the selected features | 2 | rsgen-avro-0.15.0-armv7-unknown-linux-musleabihf | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810291996 | | error: failed to push some refs to 'github.com:cargo-bins/cargo-quickinstall.git' | 2 | build-popular-package-failure | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810006866 | | error: could not compile `termion` (lib) due to 5 previous errors | 2 | natls-2.1.14-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810122999 | | error: `yield` can only be used in `#[coroutine]` closures, or `gen` blocks | 1 | drone-0.14.0-x86_64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809917129 | | error: could not compile `drone` (lib) due to 26 previous errors | 1 | drone-0.14.0-x86_64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809917129 | | error: could not compile `socket2` (lib) due to 1 previous error | 1 | spotify-tui-0.25.0-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810072070 | | error: failed to run custom build command for `clippy ` | 1 | clippy-0.0.302-x86_64-apple-darwin | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809771163 | | error: failed to run custom build command for `sha1-asm ` | 1 | starship-1.20.1-aarch64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810138319 | | error: could not compile `lurk-cli` (lib) due to 1 previous error | 1 | lurk-cli-0.3.6-x86_64-unknown-linux-musl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809926443 | | error: failed to run custom build command for `ffmpeg-sys-next ` | 1 | blissify-0.5.0-x86_64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809855029 | | error: failed to run custom build command for `libxml ` | 1 | build-popular-package-hurl-5.0.1-aarch64-unknown-linux-musl-vendored-openssl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810241945 | | error: there is nothing to install in `tp-note v1.21.16`, because it has no binaries | 1 | tp-note-1.21.16-aarch64-unknown-linux-musl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810255221 | | error: could not compile `ring` (build script) due to 1 previous error | 1 | cloak-0.2.0-aarch64-apple-darwin | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809811427 | | error: failed to run custom build command for `nettle-sys ` | 1 | sequoia-sq-0.38.0-x86_64-apple-darwin | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809789007 | | error: could not compile `lfs-core` (lib) due to 4 previous errors | 1 | dysk-2.9.1-x86_64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810015176 | | failed to select a version for the requirement `clap = "~2.11.2"` | 1 | cargo-count-0.2.4-x86_64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810038142 | | error: there is nothing to install in `watchexec v4.1.0`, because it has no binaries | 1 | watchexec-4.1.0-aarch64-unknown-linux-musl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810212961 | | error: could not compile `gix-credentials` (lib) due to 2 previous errors | 1 | cargo-crev-0.25.9-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810078540 | | error: there is nothing to install in `tauri-bundler v1.6.1`, because it has no binaries | 1 | build-popular-package-tauri-bundler-1.6.1-armv7-unknown-linux-gnueabihf-native-tls-vendored | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809487033 | | error: failed to run custom build command for `psm ` | 1 | jql-7.1.13-aarch64-unknown-linux-musl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810204806 | | error: linking with `/home/runner/.cache/cargo-zigbuild/0.19.1/zigcc-aarch64-unknown-linux-gnu.2.17-4c50.sh` failed: exit status: 1 | 1 | nitrocli-0.4.1-aarch64-unknown-linux-gnu | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810188732 | | error: there is nothing to install in `gptman v1.1.2`, because it has no binaries | 1 | gptman-1.1.2-x86_64-unknown-linux-musl | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10809934932 | | error: failed to run custom build command for `hoedown ` | 1 | cargo-script-0.2.8-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810092384 | | error: could not compile `rustc-serialize` (lib) due to 1 previous error | 1 | project_init-3.1.23-aarch64-pc-windows-msvc | https://github.com/cargo-bins/cargo-quickinstall/actions/runs/10810137594 |

[NobodyXu]

Wow this really looks great!

Looking at the message:

error: there is nothing to install in watchexec v4.1.0, because it has no binaries

I think we could filter out crates without any binary provided.

We could filter it out efficiently using crates-io db dump, given that a crate with binary is likely going to keep providing a binary, and if a crate just adds a binary to it, then it might have to wait for 1d.

https://crates.io/api/v1/crates/ripgrep/14.1.0 has a bin_names field for all binaries, unfortunately it's not available in sparse index.

The next best thing we can do, is to download the crate file and check if it has a binary.

Because if users requests a non-binary crate to be built, we would waste time building all of the dependencies and the crate, until cargo-install finds that no binary is available.

That'd be a waste of time, it's bad because we have limited free runner time on GitHub.

[alsuren]

We could filter it out efficiently using crates-io db dump, given that a crate with binary is likely going to keep providing a binary, and if a crate just adds a binary to it, then it might have to wait for 1d.

It turns out we're only building these crates because they're in popular-crates.txt (

$ git grep gptma HEAD~50 -- .
HEAD~50:popular-crates.txt:162:gptman

)

None of them are present in our influxdb logs.

If your get_popular_crates() thing filters out crates with no binaries then we should be safe, and this should stop happening on its own.

That'd be a waste of time, it's bad because we have limited free runner time on GitHub.

have we ever actually hit this limit? I will have another go at grouping things, but I suspect this case is still only a fraction of the openssl-sys errors.

[NobodyXu]

If your get_popular_crates() thing filters out crates with no binaries then we should be safe, and this should stop happening on its own.

Yeah get_crates_io_popular_crates only keeps crates with binaries, otherwise serde, syn, proc-macro2 would be the top crates to be built.

[NobodyXu]

have we ever actually hit this limit? I will have another go at grouping things, but I suspect this case is still only a fraction of the openssl-sys errors.

IIRC it happens once before, not very common but occasionally does happen