internal error: entered unreachable code: Expected at least one stalled top-level await

[panva]

Version: Deno 1.44.0

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at [https://github.com/denoland/deno/issues/new.](https://github.com/denoland/deno/issues/new.?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: linux x86_64
Version: 1.44.0
Args: ["deno", "run", "--allow-read", "--allow-net", "--import-map", "tap/import_map.json", "--no-npm", "tap/run-deno.ts"]

thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba[15](https://github.com/panva/jose/actions/runs/9349170914/job/25729896095#step:8:16)001f/deno_core-0.283.0/runtime/jsruntime.rs:1856:3:
internal error: entered unreachable code: Expected at least one stalled top-level await
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: deno_core::runtime::jsruntime::JsRuntime::poll_event_loop
   3: deno_core::runtime::jsruntime::JsRuntime::run_event_loop::{{closure}}
   4: deno_runtime::worker::MainWorker::run_event_loop::{{closure}}
   5: deno::worker::CliMainWorker::run::{{closure}}
   6: deno::tools::run::run_script::{{closure}}
   7: deno::spawn_subcommand::{{closure}}
   8: <deno_unsync::task::MaskFutureAsSend<F> as core::future::future::Future>::poll
   9: tokio::runtime::task::raw::poll
  10: deno::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Happened since the latest two releases after running npm upgrade (but never again after) and intermittenly in CI after denoland/setup-deno@v1.

Reproduction steps (What runs in CI)

gh repo clone panva/jose -- --depth=1
cd jose
npm clean-install
npm run build:deno
deno run --allow-read --allow-net --import-map tap/import_map.json --no-npm tap/run-deno.ts

---

If you run the last command again it'll pass. Running the same set of steps in a folder previously not used reproduces again. I'm assuming code cache has something to do with this.

edit:

CI runs:

1: https://github.com/panva/jose/actions/runs/9349170914/job/25729896095 2: https://github.com/panva/jose/actions/runs/9349170914/job/25732051415

Local runs:

Details

``` ➜ jose git:(main) export RUST_BACKTRACE=full ➜ jose git:(main) gh repo clone panva/jose -- --depth=1 cd jose npm clean-install npm run build:deno deno run --allow-read --allow-net --import-map tap/import_map.json --no-npm tap/run-deno.ts Cloning into 'jose'... remote: Enumerating objects: 399, done. remote: Counting objects: 100% (399/399), done. remote: Compressing objects: 100% (324/324), done. remote: Total 399 (delta 131), reused 162 (delta 71), pack-reused 0 Receiving objects: 100% (399/399), 319.53 KiB | 1.72 MiB/s, done. Resolving deltas: 100% (131/131), done. added 477 packages in 1s > jose@5.3.0 build:deno > npm run-script runtime-deno && find dist/deno -name '*.ts' -type f -print0 | xargs -0 sed -i.bak -e "s/\.js'/.ts'/g" -e "s/\.d'/.d.ts'/g" && npm run-script sedcleanup > jose@5.3.0 runtime-deno > npm run-script runtime-browser && mkdir -p dist/deno && cp -R src/. dist/deno && rm -R dist/deno/runtime/browser dist/deno/runtime/node > jose@5.3.0 runtime-browser > run-s runtime:clear runtime:browser:* runtime:refs > jose@5.3.0 runtime:clear > run-s -s runtime:find | xargs -0 rm -f > jose@5.3.0 runtime:browser:copy > cp ./src/runtime/browser/*.ts ./src/runtime > jose@5.3.0 runtime:refs > run-s -s runtime:find | xargs -0 sed -i.bak -e "s/'\.\.\//'\.\//g" -e "s/'\.\/\.\./'../g" && npm run-script sedcleanup > jose@5.3.0 sedcleanup > find . -name '*.bak' -type f -print0 | xargs -0 rm -f > jose@5.3.0 sedcleanup > find . -name '*.bak' -type f -print0 | xargs -0 rm -f ============================================================ Deno has panicked. This is a bug in Deno. Please report this at https://github.com/denoland/deno/issues/new. If you can reliably reproduce this panic, include the reproduction steps and re-run with the RUST_BACKTRACE=1 env var set and include the backtrace in your report. Platform: macos aarch64 Version: 1.44.0 Args: ["deno", "run", "--allow-read", "--allow-net", "--import-map", "tap/import_map.json", "--no-npm", "tap/run-deno.ts"] thread 'main' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.283.0/runtime/jsruntime.rs:1856:3: internal error: entered unreachable code: Expected at least one stalled top-level await stack backtrace: 0: 0x103a7a8c4 - ::fmt::h0aa20ca08aeb683c 1: 0x102c1c0ec - core::fmt::write::h168dbafcf35bac68 2: 0x103a4e0e8 - std::io::Write::write_fmt::hdb0dd3f09dcf2281 3: 0x103a7ca00 - std::sys_common::backtrace::print::h57b289e4b951ee17 4: 0x103a7c72c - std::panicking::default_hook::{{closure}}::h783b6c512154ec65 5: 0x103a7c2ec - std::panicking::default_hook::hcdfa9e1e0f234a4f 6: 0x102ac9ca8 - deno::setup_panic_hook::{{closure}}::hd392e59fb635e42f 7: 0x103a7d394 - std::panicking::rust_panic_with_hook::h9aea678ca49d64cf 8: 0x103a7d128 - std::panicking::begin_panic_handler::{{closure}}::ha16c3377e66deceb 9: 0x103a7d0b8 - std::sys_common::backtrace::__rust_end_short_backtrace::hea8fdda1ea8a4c0e 10: 0x103a7d0ac - _rust_begin_unwind 11: 0x102c1a494 - core::panicking::panic_fmt::h1cb43b60f5788132 12: 0x102d7e468 - deno_core::runtime::jsruntime::JsRuntime::poll_event_loop::heafc8bf4f70f6828 13: 0x10291b084 - deno_core::runtime::jsruntime::JsRuntime::run_event_loop::{{closure}}::hac84394b34a5e36f 14: 0x10279bddc - deno_runtime::worker::MainWorker::run_event_loop::{{closure}}::ha2fb5faf2fb36603 15: 0x102aaa77c - deno::worker::CliMainWorker::run::{{closure}}::h9dbcee303be3cd75 16: 0x102a78cb8 - deno::tools::run::run_script::{{closure}}::h7fa9c49ceed51745 17: 0x102abe4cc - deno::spawn_subcommand::{{closure}}::hbada03c742427b80 18: 0x1029019ec - as core::future::future::Future>::poll::h89f96388045c74d2 19: 0x102872230 - tokio::runtime::task::raw::poll::hc17266e3a0356e1a 20: 0x102acb31c - deno::main::hf77d18a9b576d6fd 21: 0x1027cc89c - std::sys_common::backtrace::__rust_begin_short_backtrace::h4e9d3f0b60c8b04c 22: 0x102b2fdac - _main ```

[panva]

Tested again with https://github.com/denoland/deno/releases/tag/v1.44.1, issue still persists.

[bartlomieju]

Looks like this will be fixed by https://github.com/denoland/deno/pull/24128.

[panva]

24128 (https://github.com/denoland/deno/releases/tag/v1.44.2) did not fix this issue

[bartlomieju]

@panva we've been investigating this issue with @nathanwhit and so far we established that the problem appeared when we upgraded from V8 12.4 to V8 12.6. The investigation leads us to believe that it might be a V8 bug, we're trying to strip down the reproduction and validate the idea.

[nathanwhit]

I was able to minimize the reproduction and I'm now fairly sure this is a V8 bug. You can see the reproducer here: https://github.com/nathanwhit/v8-async-module-hang (that relies on part of V8). There's an equivalent reproducer for deno here: https://github.com/nathanwhit/deno-async-module-hang (there are a couple tiny differences with the above due to some differences in event loops).

I've reported this upstream (here), and I'll try to update here once I hear back

[lucacasonato]

The reproduction reproduces in all versions of Deno down to at least 1.15.0 - seems to be a long standing bug in V8.

[panva]

How come in the reproduction case you only encounter it the first time the code runs in a given directory? And only since a couple releases ago?

[lucacasonato]

@panva It's timing dependant - I can reproduce it every time on an M1 MacBook Air.

Anyway, it is a confirmed upstream bug - I'm working with Shu from V8 to figure out a fix.

[bartlomieju]

The fix has landed in https://chromium-review.googlesource.com/c/v8/v8/+/5631143. We'll most likely will need to wait until the next minor release as it will require to update V8.

[bartlomieju]

The bug was fixed in V8, just waiting for it to be available in one of the lkgr branches.

[panva]

I believe this has been resolved with 1.46.0

[bartlomieju]

That's right, the fix should have been included in V8 12.8 which Deno 1.46 ships with. Thanks for providing the repro.