ICE with rustc 1.64.0-nightly (27eb6d701 2022-07-04) running on x86_64-unknown-linux-gnu

[andrewdavidmackenzie]

Maintainer's Note

This is a bug with recent nightly versions as documented here, and likely due to MIR inlining in https://github.com/rust-lang/rust/pull/91743. The last known working commit is nightly-x86_64-unknown-linux-gnu installed - rustc 1.64.0-nightly (7425fb293 2022-06-30). See this comment further below for a deeper description of the issue. This is now a tracking issue for https://github.com/rust-lang/rust/issues/98833.

The original post is unmodified below.

---

Description

My projects nightly cron Github Action build has started failing when compiling minimal-lexical recently, I presume due to an update to rustc nightly...

Here is a failing build log:

https://github.com/andrewdavidmackenzie/flow/runs/7201189157?check_suite_focus=true#step:12:229

It seems to be deep in my dependency tree via wax -> nom -> minimal-lexical, and I have wax on 0.5.0 which is the latest, so not sure there is much I can do on my end?

[Alexhuszagh]

Description

My projects nightly cron Github Action build has started failing when compiling minimal-lexical recently, I presume due to an update to rustc nightly...

Here is a failing build log:

https://github.com/andrewdavidmackenzie/flow/runs/7201189157?check_suite_focus=true#step:12:229

It seems to be deep in my dependency tree via wax -> nom -> minimal-lexical, and I have wax on 0.5.0 which is the latest, so not sure there is much I can do on my end?

I'm a little bit swamped with the release of cross v0.2.3 but I'll be working on this shortly. Sorry for the delay, I've been backlogged.

[andrewdavidmackenzie]

Hi, any progress? Can I help? Have a bunch of builds from dependabot piling up, but all failing due to this.

[Alexhuszagh]

I think this is more than just a bug in lexical, but I'll take a deep dive. We're running into this in a few other libraries I maintain.

[Alexhuszagh]

Can you try with CARGO_INCREMENTAL=0, just as a test? Running cargo clean on a relatively complex build system fixed ICEs in one library I maintain, so I think this might be an issue with incremental compilation. This doesn't reproduce either just building lexical alone. I'm using:

18:57 $ cargo +nightly --version
cargo 1.64.0-nightly (b1dd22e66 2022-07-09)

And then cargo +nightly build and cargo +nightly test both pass. I've got a reproducing failure in cross, so we should be able to bisect the version that caused this.

[Alexhuszagh]

This isn't a lexical issue: this is the issue in question https://github.com/rust-lang/rust/issues/98833. Specifically, this comment documents the likely offending commit.

Bisection says

searched nightlies: from nightly-2022-06-29 to nightly-2022-07-03 regressed nightly: nightly-2022-07-03 searched commit range: https://github.com/rust-lang/rust/compare/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f...f2d93935ffba3ab9d7ccb5300771a2d29b4c8bf3 regressed commit: https://github.com/rust-lang/rust/commit/0075bb4fad68e64b6d1be06bf2db366c30bc75e1 bisected with cargo-bisect-rustc v0.6.1

So, https://github.com/rust-lang/rust/pull/91743

I'll keep this open as a tracking issue, but this isn't a bug in lexical, and is an upstream issue. Hopefully you find a working solution, such as using an older nightly in the meantime.

[andrewdavidmackenzie]

It's failing in my GH Action. I do have cargo cached. So I tried changing the key to do an effective reset of the cache. No change.

[Alexhuszagh]

I'll try building your project with various nightlies since it seemed that nightly-2022-06-29-x86_64-unknown-linux-gnu did not work, which should be before the regression based on this workflow. I'll try using cargo-bisect-rustc to see where the issue is. Even if I can't fix this (ICEs are generally not an issue in libraries unless they do something very unusual, which lexical does not do).

(Note that this may take a while, your package doesn't compile very quickly and I believe I need the full context to check this).

[Alexhuszagh]

FYI: If you want to help (this will likely be filed as a bug on Rust itself), using cargo-bisect-rustc with all the necessary dependencies installed and the failing action would be very useful. You would be using a custom script, and the start and end points can be toggled as per the description. Once I have more information on what's failing and the dependencies, I can help you get started with using the tool.

[Alexhuszagh]

I'm using a Fedora 36 host machine, and ran the following:

$ rustup toolchain install nightly-2022-06-29-x86_64-unknown-linux-gnu
$ cargo +nightly-2022-06-29 build --manifest-path flowc/Cargo.toml

I also tried it with nightly-2022-07-17-x86_64-unknown-linux-gnu and nothing reproduces, so this seems very difficult to debug. If you can get a reproducible case, that would likely be necessary for me to bisect this.

[andrewdavidmackenzie]

Battling this now, is strange. It maybe related to coverage and the compiler flags needed for that. I will try to narrow it down and post back here before I leave for a weeks vacation tonight....

[andrewdavidmackenzie]

Here is the crash trace from GH Actions (https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true), which includes the -C instrument-coverage., which is added in the GH Action via echo RUSTFLAGS="-C instrument-coverage" >> "$GITHUB_ENV".

I'll see if I can remove that and see what happens.

Compiling minimal-lexical v0.2.1
thread 'rustc' panicked at 'No counters provided the source_hash for used function: Instance { def: Item(WithOptConstParam { did: DefId(0:372 ~ minimal_lexical[6522]::stackvec::{impl#0}::capacity), const_param_did: None }), substs: [] }', compiler/rustc_codegen_ssa/src/coverageinfo/map.rs:156:9
stack backtrace:
   0:     0x7f505d412a80 - std::backtrace_rs::backtrace::libunwind::trace::h972caad916e73545
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   1:     0x7f505d412a80 - std::backtrace_rs::backtrace::trace_unsynchronized::he590498[78](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:79)fe5a05d
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f505d412a80 - std::sys_common::backtrace::_print_fmt::he4a91f9bcfad9b40
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:66:5
   3:     0x7f505d412a80 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h51433dc001920472
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:45:22
   4:     0x7f505d46c05c - core::fmt::write::hc9dbd37d69b2c204
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/core/src/fmt/mod.rs:1198:17
   5:     0x7f505d404085 - std::io::Write::write_fmt::h6b2550ce8adb9e04
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/io/mod.rs:1672:15
   6:     0x7f505d415711 - std::sys_common::backtrace::_print::h006829bd22a5a4ee
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:48:5
   7:     0x7f505d415711 - std::sys_common::backtrace::print::h0f4d319136ab4456
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:35:9
   8:     0x7f505d415711 - std::panicking::default_hook::{{closure}}::h5b3cdff51fbe7401
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:295:22
   9:     0x7f505d4153e3 - std::panicking::default_hook::hdc1d8baf28b4ffd7
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:314:9
  10:     0x7f505dc43c94 - rustc_driver[9c4cb0a8f012e98d]::DEFAULT_HOOK::{closure#0}::{closure#0}
  11:     0x7f505d415ee6 - std::panicking::rust_panic_with_hook::h80e138cc00203db9
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:702:17
  12:     0x7f505d415d37 - std::panicking::begin_panic_handler::{{closure}}::hcdb59a5644417720
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:588:13
  13:     0x7f505d412f64 - std::sys_common::backtrace::__rust_end_short_backtrace::h780df157186bbb24
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:138:18
  14:     0x7f505d415a62 - rust_begin_unwind
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:584:5
  15:     0x7f505d3d9d03 - core::panicking::panic_fmt::h986ac7ffce6fbfc6
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/core/src/panicking.rs:142:14
  16:     0x7f505e77af6a - <rustc_codegen_ssa[6a1966fcaaca95a1]::coverageinfo::map::FunctionCoverage>::get_expressions_and_counter_regions
  17:     0x7f505dd6aec8 - rustc_codegen_llvm[c56b3962df227c03]::coverageinfo::mapgen::finalize
  18:     0x7f505efcfc2e - rustc_codegen_llvm[c56b3962df227c03]::base::compile_codegen_unit::module_codegen
  19:     0x7f505fe6d0e9 - <rustc_query_system[9708906f6cec1c7d]::dep_graph::graph::DepGraph<rustc_middle[161a1a9e9c329430]::dep_graph::dep_node::DepKind>>::with_task::<rustc_middle[161a1a9e9c329430]::ty::context::TyCtxt, rustc_span[55267a27b309ff44]::symbol::Symbol, rustc_codegen_ssa[6a1966fcaaca95a1]::ModuleCodegen<rustc_codegen_llvm[c56b3962df227c03]::ModuleLlvm>>
  20:     0x7f505fe9c049 - rustc_codegen_llvm[c56b3962df227c03]::base::compile_codegen_unit
  21:     0x7f505fe70469 - rustc_codegen_ssa[6a1966fcaaca95a1]::base::codegen_crate::<rustc_codegen_llvm[c56b3962df227c03]::LlvmCodegenBackend>
  22:     0x7f505feb3381 - <rustc_codegen_llvm[c56b3962df227c03]::LlvmCodegenBackend as rustc_codegen_ssa[6a1966fcaaca95a1]::traits::backend::CodegenBackend>::codegen_crate
  23:     0x7f505fe34f17 - <rustc_session[f4935b89197e8a0c]::session::Session>::time::<alloc[43276b076c5fce87]::boxed::Box<dyn core[8474d0da3e68775f]::any::Any>, rustc_interface[c134a4e7f0d41e67]::passes::start_codegen::{closure#0}>
  24:     0x7f505fe318a3 - <rustc_interface[c134a4e7f0d41e67]::passes::QueryContext>::enter::<<rustc_interface[c134a4e7f0d41e67]::queries::Queries>::ongoing_codegen::{closure#0}::{closure#0}, core[8474d0da3e68775f]::result::Result<alloc[43276b076c5fce87]::boxed::Box<dyn core[8474d0da3e68775f]::any::Any>, rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  25:     0x7f505fe27e63 - <rustc_interface[c134a4e7f0d41e67]::queries::Queries>::ongoing_codegen
  26:     0x7f505fdef881 - <rustc_interface[c134a4e7f0d41e67]::interface::Compiler>::enter::<rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}::{closure#2}, core[8474d0da3e68775f]::result::Result<core[8474d0da3e68775f]::option::Option<rustc_interface[c134a4e7f0d41e67]::queries::Linker>, rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  27:     0x7f505fdeb9cf - rustc_span[55267a27b309ff44]::with_source_map::<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_interface[c134a4e7f0d41e67]::interface::create_compiler_and_run<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#1}>
  28:     0x7f505fe080a0 - rustc_interface[c134a4e7f0d41e67]::interface::create_compiler_and_run::<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>
  29:     0x7f505fe1c0a2 - <scoped_tls[a5183b1e9084a86b]::ScopedKey<rustc_span[55267a27b309ff44]::SessionGlobals>>::set::<rustc_interface[c134a4e7f0d41e67]::interface::run_compiler<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  30:     0x7f505fdedeff - std[ec26f314c4b749cb]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[c134a4e7f0d41e67]::util::run_in_thread_pool_with_globals<rustc_interface[c134a4e7f0d41e67]::interface::run_compiler<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  31:     0x7f505fe08519 - <<std[ec26f314c4b749cb]::thread::Builder>::spawn_unchecked_<rustc_interface[c134a4e7f0d41e67]::util::run_in_thread_pool_with_globals<rustc_interface[c134a4e7f0d41e67]::interface::run_compiler<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>::{closure#1} as core[8474d0da3e68775f]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  32:     0x7f505d41f933 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc16ea7f50c9c25c4
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/alloc/src/boxed.rs:1935:9
  33:     0x7f505d41f933 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h2[79](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:80)c120d7b8bf0d7
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/alloc/src/boxed.rs:1935:9
  34:     0x7f505d41f933 - std::sys::unix::thread::Thread::new::thread_start::h28d4581e84daad0c
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys/unix/thread.rs:108:17
  35:     0x7f505d335609 - start_thread
  36:     0x7f505d258133 - clone
  37:                0x0 - <unknown>
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.64.0-nightly (d68e7ebc3 2022-07-20) running on x86_64-unknown-linux-gnu
note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C instrument-coverage
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
end of query stack
error: could not compile `minimal-lexical`
warning: build failed, waiting for other jobs to finish...
thread 'rustc' panicked at 'No counters provided the source_hash for used function: Instance { def: Item(WithOptConstParam { did: DefId(0:5017 ~ gimli[8058]::constants::{impl#92}::ne), const_param_did: None }), substs: [] }', compiler/rustc_codegen_ssa/src/coverageinfo/map.rs:156:9
stack backtrace:
   0:     0x7f50c6617a[80](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:81) - std::backtrace_rs::backtrace::libunwind::trace::h972caad916e73545
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   1:     0x7f50c6617a80 - std::backtrace_rs::backtrace::trace_unsynchronized::he59049878fe5a05d
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f50c6617a80 - std::sys_common::backtrace::_print_fmt::he4a91f9bcfad9b40
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:66:5
   3:     0x7f50c6617a80 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h51433dc001920472
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:45:22
   4:     0x7f50c667105c - core::fmt::write::hc9dbd37d69b2c204
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/core/src/fmt/mod.rs:1198:17
   5:     0x7f50c6609085 - std::io::Write::write_fmt::h6b2550ce8adb9e04
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/io/mod.rs:1672:15
   6:     0x7f50c661a711 - std::sys_common::backtrace::_print::h006829bd22a5a4ee
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:48:5
   7:     0x7f50c661a711 - std::sys_common::backtrace::print::h0f4d319136ab4456
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:35:9
   8:     0x7f50c661a711 - std::panicking::default_hook::{{closure}}::h5b3cdff51fbe7401
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:295:22
   9:     0x7f50c661a3e3 - std::panicking::default_hook::hdc1d8baf28b4ffd7
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:314:9
  10:     0x7f50c6e48c94 - rustc_driver[9c4cb0a8f012e98d]::DEFAULT_HOOK::{closure#0}::{closure#0}
  11:     0x7f50c661aee6 - std::panicking::rust_panic_with_hook::h80e138cc00203db9
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:702:17
  12:     0x7f50c661ad37 - std::panicking::begin_panic_handler::{{closure}}::hcdb59a5644417720
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:588:13
  13:     0x7f50c6617f64 - std::sys_common::backtrace::__rust_end_short_backtrace::h780df157186bbb24
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:138:18
  14:     0x7f50c661aa62 - rust_begin_unwind
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:584:5
  15:     0x7f50c65ded03 - core::panicking::panic_fmt::h986ac7ffce6fbfc6
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/core/src/panicking.rs:142:14
  16:     0x7f50c797ff6a - <rustc_codegen_ssa[6a1966fcaaca95a1]::coverageinfo::map::FunctionCoverage>::get_expressions_and_counter_regions
  17:     0x7f50c6f6fec8 - rustc_codegen_llvm[c56b3962df227c03]::coverageinfo::mapgen::finalize
  18:     0x7f50c[81](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:82)d4c2e - rustc_codegen_llvm[c56b3962df227c03]::base::compile_codegen_unit::module_codegen
  19:     0x7f50c90720e9 - <rustc_query_system[9708906f6cec1c7d]::dep_graph::graph::DepGraph<rustc_middle[161a1a9e9c329430]::dep_graph::dep_node::DepKind>>::with_task::<rustc_middle[161a1a9e9c329430]::ty::context::TyCtxt, rustc_span[55267a27b309ff44]::symbol::Symbol, rustc_codegen_ssa[6a1966fcaaca95a1]::ModuleCodegen<rustc_codegen_llvm[c56b3962df227c03]::ModuleLlvm>>
  20:     0x7f50c90a1049 - rustc_codegen_llvm[c56b3962df227c03]::base::compile_codegen_unit
  21:     0x7f50c9075469 - rustc_codegen_ssa[6a1966fcaaca95a1]::base::codegen_crate::<rustc_codegen_llvm[c56b3962df227c03]::LlvmCodegenBackend>
  22:     0x7f50c90b[83](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:84)81 - <rustc_codegen_llvm[c56b3962df227c03]::LlvmCodegenBackend as rustc_codegen_ssa[6a1966fcaaca95a1]::traits::backend::CodegenBackend>::codegen_crate
  23:     0x7f50c9039f17 - <rustc_session[f4935b89197e8a0c]::session::Session>::time::<alloc[43276b076c5fce87]::boxed::Box<dyn core[[84](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:85)74d0da3e68775f]::any::Any>, rustc_interface[c134a4e7f0d41e67]::passes::start_codegen::{closure#0}>
  24:     0x7f50c90368a3 - <rustc_interface[c134a4e7f0d41e67]::passes::QueryContext>::enter::<<rustc_interface[c134a4e7f0d41e67]::queries::Queries>::ongoing_codegen::{closure#0}::{closure#0}, core[8474d0da3e68775f]::result::Result<alloc[43276b076c5fce87]::boxed::Box<dyn core[8474d0da3e68775f]::any::Any>, rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  25:     0x7f50c902ce63 - <rustc_interface[c134a4e7f0d41e67]::queries::Queries>::ongoing_codegen
  26:     0x7f50c8ff4881 - <rustc_interface[c134a4e7f0d41e67]::interface::Compiler>::enter::<rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}::{closure#2}, core[8474d0da3e68775f]::result::Result<core[8474d0da3e68775f]::option::Option<rustc_interface[c134a4e7f0d41e67]::queries::Linker>, rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  27:     0x7f50c8ff09cf - rustc_span[55267a27b309ff44]::with_source_map::<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_interface[c134a4e7f0d41e67]::interface::create_compiler_and_run<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#1}>
  28:     0x7f50c900d0a0 - rustc_interface[c134a4e7f0d41e67]::interface::create_compiler_and_run::<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>
  29:     0x7f50c90210a2 - <scoped_tls[a5183b1e9084a[86](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:87)b]::ScopedKey<rustc_span[55267a27b309ff44]::SessionGlobals>>::set::<rustc_interface[c134a4e7f0d41e67]::interface::run_compiler<core[8474d0da3e6[87](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:88)75f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  30:     0x7f50c8ff2eff - std[ec26f314c4b749cb]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[c134a4e7f0d41e67]::util::run_in_thread_pool_with_globals<rustc_interface[c134a4e7f0d41e67]::interface::run_compiler<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>
  31:     0x7f50c[90](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:91)0d519 - <<std[ec26f314c4b749cb]::thread::Builder>::spawn_unchecked_<rustc_interface[c134a4e7f0d41e67]::util::run_in_thread_pool_with_globals<rustc_interface[c134a4e7f0d41e67]::interface::run_compiler<core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>, rustc_driver[9c4cb0a8f012e98d]::run_compiler::{closure#1}>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>::{closure#0}, core[8474d0da3e68775f]::result::Result<(), rustc_errors[27ce1cbc1ec9d60f]::ErrorGuaranteed>>::{closure#1} as core[8474d0da3e68775f]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  32:     0x7f50c6624933 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc16ea7f50c9c25c4
                               at /rustc/d68e7ebc38cb42b8b2373[92](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:93)b28045edeec761503/library/alloc/src/boxed.rs:1[93](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:94)5:9
  33:     0x7f50c6624933 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h279c120d7b8bf0d7
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/alloc/src/boxed.rs:1935:9
  34:     0x7f50c6624933 - std::sys::unix::thread::Thread::new::thread_start::h28d4581e84daad0c
                               at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys/unix/thread.rs:108:17
  35:     0x7f50c653a609 - start_thread
  36:     0x7f50c645d133 - clone
  37:                0x0 - <unknown>
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.64.0-nightly (d68e7ebc3 2022-07-20) running on x86_64-unknown-linux-gnu
note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C instrument-coverage
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
end of query stack
error: could not compile `gimli`
error: failed to compile `flowc v0.60.0 (/home/runner/work/flow/flow/flowc)`, intermediate artifacts can be found at `/home/runner/work/flow/flow/target`
Error: Process completed with exit code [101](https://github.com/andrewdavidmackenzie/flow/runs/7464889220?check_suite_focus=true#step:12:102).

[andrewdavidmackenzie]

This seems to have fixed itself while I was on Vacations :-) I assume by an update to rust nightly?