search

oxidecomputer / omicron

Omicron: Oxide control plane
Mozilla Public License 2.0
244 stars 38 forks source link

flaky build failure: "remote wall time is too far ahead" #962

Open davepacheco opened 2 years ago

davepacheco commented 2 years ago

Here's where we saw it: https://github.com/oxidecomputer/omicron/runs/6133927246?check_suite_focus=true#step:11:1956

error: failed to run custom build command for `nexus-test-utils v0.1.0 (/Users/runner/work/omicron/omicron/nexus/test-utils)`

Caused by:
  process didn't exit successfully: `/Users/runner/work/omicron/omicron/target/debug/build/nexus-test-utils-08a85905696fbfd8/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=build.rs
  cargo:rerun-if-changed=../../common/src/sql/dbinit.sql
  cargo:rerun-if-changed=../../tools/cockroachdb_checksums
  cargo:rerun-if-changed=../../tools/cockroachdb_version

  --- stderr
  Apr 22 19:42:29.902 INFO cockroach temporary directory: /tmp/omicron_tmp/.tmpCY01Eg
  Apr 22 19:42:29.902 INFO cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store /Users/runner/work/omicron/omicron/target/debug/build/nexus-test-utils-4b01e832e8f96d40/out/crdb-base --listen-addr 127.0.0.1:0 --listening-url-file /tmp/omicron_tmp/.tmpCY01Eg/listen-url
  Apr 22 19:42:42.034 INFO cockroach pid: 7411
  Apr 22 19:42:42.034 INFO cockroach listen URL: postgresql://root@127.0.0.1:49430/omicron?sslmode=disable
  Apr 22 19:42:42.034 INFO cockroach: populating
  thread 'main' panicked at 'failed to populate database: populate

  Caused by:
      0: populating Omicron database
      1: db error: ERROR: polling for queued jobs to complete: poll-show-jobs: remote wall time is too far ahead (1.732092s) to be trustworthy
      2: ERROR: polling for queued jobs to complete: poll-show-jobs: remote wall time is too far ahead (1.732092s) to be trustworthy', /Users/runner/work/omicron/omicron/test-utils/src/dev/mod.rs:150:35
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
  WARN: temporary directory leaked: /tmp/omicron_tmp/.tmpCY01Eg
warning: build failed, waiting for other jobs to finish...
error: build failed
Error: Process completed with exit code 101.

This message appears to come from CockroachDB, which has a poll-show-jobs thing and a component with this error message. What's weird is that this is a single-node CockroachDB cluster and there's only one system here. My first thought was maybe the system clock jumped during the test, but that seems unlikely. I wonder if this is a symptom of CPU starvation due to GitHub Actions workers being starved. That is: maybe CockroachDB gets a timestamp on the client, gets one from the server, and compares them, and hits this message when they're too far apart. On a single system, you could still fail that check if you were stuck off-CPU for a while between the calls to get timestamps.

davepacheco commented 2 years ago

We saw this again today: https://github.com/oxidecomputer/omicron/pull/1341/checks?check_run_id=7308022526

This time it was in buildomat. That rules out the GitHub actions CPU starvation idea. I guess it's conceivable that general CPU starvation could cause it.

In case we lose the logs:


3621 | 2022-07-12T19:11:02.253Z | Running `/home/build/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/bin/rustc --crate-name propolis_client --edition=2018 /home/build/.cargo/git/checkouts/propolis-12517f89d3d9f483/23a38f4/client/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C metadata=b2b20d4625ea4aa5 -C extra-filename=-b2b20d4625ea4aa5 --out-dir /work/oxidecomputer/omicron/target/debug/deps -L dependency=/work/oxidecomputer/omicron/target/debug/deps --extern crucible=/work/oxidecomputer/omicron/target/debug/deps/libcrucible-c84d5637ef5976bc.rmeta --extern reqwest=/work/oxidecomputer/omicron/target/debug/deps/libreqwest-a2c14320e2f28770.rmeta --extern ring=/work/oxidecomputer/omicron/target/debug/deps/libring-cc7a691315ce3505.rmeta --extern schemars=/work/oxidecomputer/omicron/target/debug/deps/libschemars-aab3ddbf8e639a29.rmeta --extern serde=/work/oxidecomputer/omicron/target/debug/deps/libserde-f9ef5320ccd19e17.rmeta --extern serde_json=/work/oxidecomputer/omicron/target/debug/deps/libserde_json-728098272d8b83c0.rmeta --extern slog=/work/oxidecomputer/omicron/target/debug/deps/libslog-58d05147ddff7e23.rmeta --extern thiserror=/work/oxidecomputer/omicron/target/debug/deps/libthiserror-4a10331472d1684e.rmeta --extern uuid=/work/oxidecomputer/omicron/target/debug/deps/libuuid-c2371772954dc660.rmeta --cap-lints allow -D warnings -L native=/usr/lib/amd64 -L native=/work/oxidecomputer/omicron/target/debug/build/ring-ac6ac2a5559bb77a/out -L native=/usr/lib/amd64`
-- | -- | --
3622 | 2022-07-12T19:11:05.257Z | error: failed to run custom build command for `nexus-test-utils v0.1.0 (/work/oxidecomputer/omicron/nexus/test-utils)`
3623 | 2022-07-12T19:11:05.263Z |  
3624 | 2022-07-12T19:11:05.275Z | Caused by:
3625 | 2022-07-12T19:11:05.281Z | process didn't exit successfully: `/work/oxidecomputer/omicron/target/debug/build/nexus-test-utils-05319e4f5bfd2066/build-script-build` (exit status: 101)
3626 | 2022-07-12T19:11:05.288Z | --- stdout
3627 | 2022-07-12T19:11:05.295Z | cargo:rerun-if-changed=build.rs
3628 | 2022-07-12T19:11:05.308Z | cargo:rerun-if-changed=../../common/src/sql/dbinit.sql
3629 | 2022-07-12T19:11:05.315Z | cargo:rerun-if-changed=../../tools/cockroachdb_checksums
3630 | 2022-07-12T19:11:05.324Z | cargo:rerun-if-changed=../../tools/cockroachdb_version
3631 | 2022-07-12T19:11:05.330Z |  
3632 | 2022-07-12T19:11:05.337Z | --- stderr
3633 | 2022-07-12T19:11:05.344Z | Jul 12 19:11:03.767 INFO cockroach temporary directory: /var/tmp/omicron_tmp/.tmpNKFQYZ
3634 | 2022-07-12T19:11:05.352Z | Jul 12 19:11:03.768 INFO cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/work/oxidecomputer/omicron/target/debug/build/nexus-test-utils-357b0cb84efd2e5b/out/crdb-base,ballast-size=0 --listen-addr 127.0.0.1:0 --listening-url-file /var/tmp/omicron_tmp/.tmpNKFQYZ/listen-url
3635 | 2022-07-12T19:11:05.367Z | Jul 12 19:11:06.046 INFO cockroach pid: 3564
3636 | 2022-07-12T19:11:05.373Z | Jul 12 19:11:06.046 INFO cockroach listen URL: postgresql://root@127.0.0.1:43777/omicron?sslmode=disable
3637 | 2022-07-12T19:11:05.380Z | Jul 12 19:11:06.046 INFO cockroach: populating
3638 | 2022-07-12T19:11:05.387Z | thread 'main' panicked at 'failed to populate database: populate
3639 | 2022-07-12T19:11:05.396Z |  
3640 | 2022-07-12T19:11:05.407Z | Caused by:
3641 | 2022-07-12T19:11:05.413Z | 0: populating Omicron database
3642 | 2022-07-12T19:11:05.421Z | 1: db error: ERROR: polling for queued jobs to complete: poll-show-jobs: remote wall time is too far ahead (1.663043185s) to be trustworthy
3643 | 2022-07-12T19:11:05.427Z | 2: ERROR: polling for queued jobs to complete: poll-show-jobs: remote wall time is too far ahead (1.663043185s) to be trustworthy', /work/oxidecomputer/omicron/test-utils/src/dev/mod.rs:151:35
3644 | 2022-07-12T19:11:05.441Z | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
3645 | 2022-07-12T19:11:05.447Z | WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
3646 | 2022-07-12T19:11:05.453Z | WARN: temporary directory leaked: /var/tmp/omicron_tmp/.tmpNKFQYZ
3647 | 2022-07-12T19:11:05.461Z | warning: build failed, waiting for other jobs to finish...
3648 | 2022-07-12T19:15:35.826Z |  
3649 | 2022-07-12T19:15:35.832Z | real    16:31.214047317
3650 | 2022-07-12T19:15:35.838Z | user  1:29:39.249598570
3651 | 2022-07-12T19:15:35.844Z | sys     14:17.226563018
3652 | 2022-07-12T19:15:35.850Z | trap        4.029516977
3653 | 2022-07-12T19:15:35.856Z | tflt       24.829313863
3654 | 2022-07-12T19:15:35.862Z | dflt       20.819714996
3655 | 2022-07-12T19:15:35.867Z | kflt        0.581077703
3656 | 2022-07-12T19:15:35.873Z | lock  3:04:11.624936065
3657 | 2022-07-12T19:15:35.879Z | slp   4:37:04.814205851
3658 | 2022-07-12T19:15:35.885Z | lat      4:47.856502453
3659 | 2022-07-12T19:15:35.890Z | stop     5:15.803805591
3660 | 2022-07-12T19:15:35.896Z | process exited: duration 1295878 ms, exit code 101
 
3661 | 2022-07-12T19:15:35.932Z | found 0 output files

[3621](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3621)  2022-07-12T19:11:02.253Z         Running `/home/build/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/bin/rustc --crate-name propolis_client --edition=2018 /home/build/.cargo/git/checkouts/propolis-12517f89d3d9f483/23a38f4/client/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C metadata=b2b20d4625ea4aa5 -C extra-filename=-b2b20d4625ea4aa5 --out-dir /work/oxidecomputer/omicron/target/debug/deps -L dependency=/work/oxidecomputer/omicron/target/debug/deps --extern crucible=/work/oxidecomputer/omicron/target/debug/deps/libcrucible-c84d5637ef5976bc.rmeta --extern reqwest=/work/oxidecomputer/omicron/target/debug/deps/libreqwest-a2c14320e2f28770.rmeta --extern ring=/work/oxidecomputer/omicron/target/debug/deps/libring-cc7a691315ce3505.rmeta --extern schemars=/work/oxidecomputer/omicron/target/debug/deps/libschemars-aab3ddbf8e639a29.rmeta --extern serde=/work/oxidecomputer/omicron/target/debug/deps/libserde-f9ef5320ccd19e17.rmeta --extern serde_json=/work/oxidecomputer/omicron/target/debug/deps/libserde_json-728098272d8b83c0.rmeta --extern slog=/work/oxidecomputer/omicron/target/debug/deps/libslog-58d05147ddff7e23.rmeta --extern thiserror=/work/oxidecomputer/omicron/target/debug/deps/libthiserror-4a10331472d1684e.rmeta --extern uuid=/work/oxidecomputer/omicron/target/debug/deps/libuuid-c2371772954dc660.rmeta --cap-lints allow -D warnings -L native=/usr/lib/amd64 -L native=/work/oxidecomputer/omicron/target/debug/build/ring-ac6ac2a5559bb77a/out -L native=/usr/lib/amd64`
[3622](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3622)  2022-07-12T19:11:05.257Z    error: failed to run custom build command for `nexus-test-utils v0.1.0 (/work/oxidecomputer/omicron/nexus/test-utils)`
[3623](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3623)  2022-07-12T19:11:05.263Z    
[3624](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3624)  2022-07-12T19:11:05.275Z    Caused by:
[3625](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3625)  2022-07-12T19:11:05.281Z      process didn't exit successfully: `/work/oxidecomputer/omicron/target/debug/build/nexus-test-utils-05319e4f5bfd2066/build-script-build` (exit status: 101)
[3626](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3626)  2022-07-12T19:11:05.288Z      --- stdout
[3627](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3627)  2022-07-12T19:11:05.295Z      cargo:rerun-if-changed=build.rs
[3628](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3628)  2022-07-12T19:11:05.308Z      cargo:rerun-if-changed=../../common/src/sql/dbinit.sql
[3629](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3629)  2022-07-12T19:11:05.315Z      cargo:rerun-if-changed=../../tools/cockroachdb_checksums
[3630](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3630)  2022-07-12T19:11:05.324Z      cargo:rerun-if-changed=../../tools/cockroachdb_version
[3631](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3631)  2022-07-12T19:11:05.330Z    
[3632](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3632)  2022-07-12T19:11:05.337Z      --- stderr
[3633](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3633)  2022-07-12T19:11:05.344Z      Jul 12 19:11:03.767 INFO cockroach temporary directory: /var/tmp/omicron_tmp/.tmpNKFQYZ
[3634](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3634)  2022-07-12T19:11:05.352Z      Jul 12 19:11:03.768 INFO cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/work/oxidecomputer/omicron/target/debug/build/nexus-test-utils-357b0cb84efd2e5b/out/crdb-base,ballast-size=0 --listen-addr 127.0.0.1:0 --listening-url-file /var/tmp/omicron_tmp/.tmpNKFQYZ/listen-url
[3635](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3635)  2022-07-12T19:11:05.367Z      Jul 12 19:11:06.046 INFO cockroach pid: 3564
[3636](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3636)  2022-07-12T19:11:05.373Z      Jul 12 19:11:06.046 INFO cockroach listen URL: postgresql://root@127.0.0.1:43777/omicron?sslmode=disable
[3637](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3637)  2022-07-12T19:11:05.380Z      Jul 12 19:11:06.046 INFO cockroach: populating
[3638](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3638)  2022-07-12T19:11:05.387Z      thread 'main' panicked at 'failed to populate database: populate
[3639](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3639)  2022-07-12T19:11:05.396Z    
[3640](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3640)  2022-07-12T19:11:05.407Z      Caused by:
[3641](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3641)  2022-07-12T19:11:05.413Z          0: populating Omicron database
[3642](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3642)  2022-07-12T19:11:05.421Z          1: db error: ERROR: polling for queued jobs to complete: poll-show-jobs: remote wall time is too far ahead (1.663043185s) to be trustworthy
[3643](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3643)  2022-07-12T19:11:05.427Z          2: ERROR: polling for queued jobs to complete: poll-show-jobs: remote wall time is too far ahead (1.663043185s) to be trustworthy', /work/oxidecomputer/omicron/test-utils/src/dev/mod.rs:151:35
[3644](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3644)  2022-07-12T19:11:05.441Z      note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[3645](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3645)  2022-07-12T19:11:05.447Z      WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
[3646](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3646)  2022-07-12T19:11:05.453Z      WARN: temporary directory leaked: /var/tmp/omicron_tmp/.tmpNKFQYZ
[3647](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3647)  2022-07-12T19:11:05.461Z    warning: build failed, waiting for other jobs to finish...
[3648](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3648)  2022-07-12T19:15:35.826Z    
[3649](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3649)  2022-07-12T19:15:35.832Z    real    16:31.214047317
[3650](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3650)  2022-07-12T19:15:35.838Z    user  1:29:39.249598570
[3651](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3651)  2022-07-12T19:15:35.844Z    sys     14:17.226563018
[3652](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3652)  2022-07-12T19:15:35.850Z    trap        4.029516977
[3653](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3653)  2022-07-12T19:15:35.856Z    tflt       24.829313863
[3654](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3654)  2022-07-12T19:15:35.862Z    dflt       20.819714996
[3655](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3655)  2022-07-12T19:15:35.867Z    kflt        0.581077703
[3656](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3656)  2022-07-12T19:15:35.873Z    lock  3:04:11.624936065
[3657](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3657)  2022-07-12T19:15:35.879Z    slp   4:37:04.814205851
[3658](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3658)  2022-07-12T19:15:35.885Z    lat      4:47.856502453
[3659](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3659)  2022-07-12T19:15:35.890Z    stop     5:15.803805591
[3660](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3660)  2022-07-12T19:15:35.896Z    process exited: duration 1295878 ms, exit code 101

[3661](https://buildomat.eng.oxide.computer/wg/0/details/01G7STYYZ5NYKEWQWBN66NET38/0efq9KYqEnfn2EwVG6ALumsGmsquO8TkkKMIIsXsa0uvq9BU/01G7STZCHY00NFMWNCWRF8HH2K#S3661)  2022-07-12T19:15:35.932Z    found 0 output files