search

mozilla / sccache

Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
Apache License 2.0
5.74k stars 542 forks source link

sccache occasionally crashes on Windows: "An existing connection was forcibly closed by the remote host. (os error 10054)" #1098

Open petamas opened 2 years ago

petamas commented 2 years ago

We use sccache on Windows to build on our CI servers, and we really like it, as it sped up our builds by a factor of 2. However, sometimes (about once a month) it crashes mid-build with the following message:

sccache: error: failed to execute compile
sccache: caused by: error reading compile response from server
sccache: caused by: Failed to read response header
sccache: caused by: An existing connection was forcibly closed by the remote host. (os error 10054)

When we last ran into this issue, we set up some logging this way:

SCCACHE_LOG="debug,sccache::config=off" // we want to disable the "could not read config file" message, as it clutters up the output
SCCACHE_ERROR_LOG="C:/...snip..."
RUST_LOG_STYLE="never"

On the client side, it resulted in only one more line (printed for every sccache invocation, including the failing one):

[2022-01-15T04:14:10Z DEBUG sccache::commands] Server sent CompileStarted

At the end of the server's logfile, we can see this (I removed the actual argument lists as they may contain sensitive data):

[2022-01-15T04:14:14Z DEBUG sccache::server] handle_client: compile
[2022-01-15T04:14:14Z DEBUG sccache::server] check_compiler: Supported compiler
[2022-01-15T04:14:14Z DEBUG sccache::server] parse_arguments: Ok: [...snip...]
[2022-01-15T04:14:14Z DEBUG sccache::compiler::compiler] [qml_NumberText_qml.cpp.obj]: get_cached_or_compile: [...snip...]
[2022-01-15T04:14:14Z DEBUG sccache::compiler::msvc] preprocess: Some(...snip...)   
[2022-01-15T04:14:14Z DEBUG tokio_reactor] adding I/O source: 27221032976
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 16
[2022-01-15T04:14:14Z DEBUG tokio_reactor] adding I/O source: 27225227266
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 2
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 15
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 24
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 23
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 27
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 22
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 3
...snip...
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 19
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 21
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 15
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 24
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 0
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 0
thread '<unnamed>' panicked at 'internal error: entered unreachable code', C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\mio-named-pipes-0.1.7\src\lib.rs:633:14
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\mio-named-pipes-0.1.7\src\lib.rs:297:46
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\mio-named-pipes-0.1.7\src\lib.rs:467:43
stack backtrace:
   0:     0x7ff60374d8b9 - <unknown>
   1:     0x7ff60376feeb - <unknown>
   2:     0x7ff603743438 - <unknown>
   3:     0x7ff603752744 - <unknown>
   4:     0x7ff603752328 - <unknown>
   5:     0x7ff603752fff - <unknown>
   6:     0x7ff603752b65 - <unknown>
   7:     0x7ff60374e19f - <unknown>
   8:     0x7ff603752b19 - <unknown>
   9:     0x7ff60376e3c0 - <unknown>
  10:     0x7ff60376e203 - <unknown>
  11:     0x7ff6033f1869 - <unknown>
  12:     0x7ff60315050b - <unknown>
  13:     0x7ff60315810b - <unknown>
  14:     0x7ff60303688c - <unknown>
  15:     0x7ff60310d4eb - <unknown>
  16:     0x7ff60311189c - <unknown>
  17:     0x7ff602fd2878 - <unknown>
  18:     0x7ff602fcade6 - <unknown>
  19:     0x7ff60323f14b - <unknown>
  20:     0x7ff603237c7b - <unknown>
  21:     0x7ff6031e4293 - <unknown>
  22:     0x7ff6031fd25a - <unknown>
  23:     0x7ff6037f4b10 - <unknown>
  24:     0x7ff6037f444e - <unknown>
  25:     0x7ff6037f23c0 - <unknown>
  26:     0x7ff6037f3ebd - <unknown>
  27:     0x7ff6037f2839 - <unknown>
  28:     0x7ffbfabba78d - _chkstk
  29:     0x7ffbfab4535a - RtlUnwindEx
  30:     0x7ff6037f26ca - <unknown>
  31:     0x7ff6037f3545 - <unknown>
  32:     0x7ff6037f389b - <unknown>
  33:     0x7ff6037f3fb9 - <unknown>
  34:     0x7ff6037f2839 - <unknown>
  35:     0x7ffbfabba70d - _chkstk
  36:     0x7ffbfab449c3 - RtlImageNtHeaderEx
  37:     0x7ffbfab466d9 - RtlRaiseException
  38:     0x7ffbf7144f38 - RaiseException
  39:     0x7ff6037f2b78 - <unknown>
  40:     0x7ff603761ba1 - <unknown>
  41:     0x7ff603761b29 - <unknown>
  42:     0x7ff603753208 - <unknown>
  43:     0x7ff6037530b6 - <unknown>
  44:     0x7ff603752b65 - <unknown>
  45:     0x7ff60374e19f - <unknown>
  46:     0x7ff603752b19 - <unknown>
  47:     0x7ff60376e3c0 - <unknown>
  48:     0x7ff60376e203 - <unknown>
  49:     0x7ff6033f07dc - <unknown>
  50:     0x7ff6033f03e7 - <unknown>
  51:     0x7ff6033e227f - <unknown>
  52:     0x7ff6033dfcad - <unknown>
  53:     0x7ff602fc599f - <unknown>
  54:     0x7ff60304e1f4 - <unknown>
  55:     0x7ff603316457 - <unknown>
  56:     0x7ff60307b86d - <unknown>
  57:     0x7ff6031b4f15 - <unknown>
  58:     0x7ff6032c8e24 - <unknown>
  59:     0x7ff6033a5767 - <unknown>
  60:     0x7ff6033554a6 - <unknown>
  61:     0x7ff603337af9 - <unknown>
  62:     0x7ff603230f8d - <unknown>
  63:     0x7ff603124081 - <unknown>
  64:     0x7ff603317c6e - <unknown>
  65:     0x7ff6032fafbb - <unknown>
  66:     0x7ff603230e6d - <unknown>
  67:     0x7ff603383360 - <unknown>
  68:     0x7ff60307bccd - <unknown>
  69:     0x7ff603367a20 - <unknown>
  70:     0x7ff6032c767e - <unknown>
  71:     0x7ff603183cfa - <unknown>
  72:     0x7ff6031fd07a - <unknown>
  73:     0x7ff6031d1344 - <unknown>
  74:     0x7ff6032518ba - <unknown>
  75:     0x7ff603443a05 - <unknown>
  76:     0x7ff60343c0bf - <unknown>
  77:     0x7ff6032bc080 - <unknown>
  78:     0x7ff6031fa377 - <unknown>
  79:     0x7ff6032be73f - <unknown>
  80:     0x7ff60325084b - <unknown>
  81:     0x7ff603073d3f - <unknown>
  82:     0x7ff603297c2e - <unknown>
  83:     0x7ff6032953c3 - <unknown>
  84:     0x7ff603188500 - <unknown>
  85:     0x7ff60327f76b - <unknown>
  86:     0x7ff60317eed2 - <unknown>
  87:     0x7ff602ffedd7 - <unknown>
  88:     0x7ff602fc1076 - <unknown>
  89:     0x7ff602fc104c - <unknown>
  90:     0x7ff603753333 - <unknown>
  91:     0x7ff602fc1037 - <unknown>
  92:     0x7ff6037f1260 - <unknown>
  93:     0x7ffbf9bc84d4 - BaseThreadInitThunk
  94:     0x7ffbfab61781 - RtlUserThreadStart
thread panicked while panicking. aborting.

Any idea what may have caused these crashes, or what other logging we could turn on to gather more information?

luser commented 2 years ago

That's deep in the bowels of mio-named-pipes. It'd be worthwhile to see if there's a newer release that we could update sccache to use.

dsanders11 commented 2 years ago

@luser, it looks like mio-named-pipes was removed as a dependency in 144eb0f1d3f6c4571cda7c30e7919933ed07612c.

I'm also seeing this crash, much more regularly, when using sccache with Chromium. It looks like the last release was a year ago, are there plans to cut a new release any time soon? At least with mio-named-pipes no longer being present the crash will either be fixed, or shift into mio itself.

cyberfoxmeow commented 1 month ago

the same error when building firefox FIREFOX_129_0_RELEASE by MozillaBuild.