search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.28k stars 4.09k forks source link

Bazel server crashing abruptly in 7.4+ #24389

Open luispadron opened 5 days ago

luispadron commented 5 days ago

Description of the bug:

Since upgrading to 7.4.0 and subsequently 7.4.1 we've been hit by a server crash during random CI (and local) builds:

Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/private/var/tmp/_bazel_build/a50428ec9d717925b6582741c513dab3/server/jvm.out')

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have not found a way to reliably reproduce this issue but were seeing about 1-3 of these per day on our ~300 machine CI fleet.

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

7.4.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

There was a GC threading issue in 7.4.0 that was also causing a server crash but that was fixed in 7.4.1 so this might be another case of that issue or something new entirely. I do not see any

Any other information, logs, or outputs that you want to share?

Attached are the Bazel server JVM log from the build it terminated on along with the .ips file generated from the crash.

jvm_logs_and_ips.zip

...

Thread 51 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x0000000000000000   x3: 0x0000000000000000
    x4: 0x0000000000000001   x5: 0x00000001777affc8   x6: 0x000000000000002e   x7: 0x0000000000000000
    x8: 0xb48a3bbbfd395f71   x9: 0xb48a3bba8a426f71  x10: 0x0000000000000002  x11: 0x00000000fffffffd
   x12: 0x0000010000000000  x13: 0x0000000000000000  x14: 0x0000000000000000  x15: 0x0000000000000000
   x16: 0x0000000000000148  x17: 0x000000020168e4e8  x18: 0x0000000000000000  x19: 0x0000000000000006
   x20: 0x00000001777b3000  x21: 0x000000000000e707  x22: 0x00000001777b30e0  x23: 0x000000000000000a
   x24: 0x0000000102ade007  x25: 0x00000001305a5e00  x26: 0x0000000000000000  x27: 0x00000000000007d0
   x28: 0x00000000ffffffff   fp: 0x00000001777b1070   lr: 0x000000018f13dc20
    sp: 0x00000001777b1050   pc: 0x000000018f1055f0 cpsr: 0x40001000
   far: 0x0000000000000000  esr: 0x56000080  Address size fault
meisterT commented 4 days ago

Do you still have the file /Users/build/.jenkins/workspace/cash-ios/ios-builder/s/c/hs_err_pid55320.log?

fmeum commented 4 days ago

Another JVM crash on macOS that may or may not be related: https://github.com/bazelbuild/bazel/issues/23497

meisterT commented 4 days ago

It may just be that the best path forward is to use an upgraded version of the JDK in the hope that this will fix the crashes.

While looking at this, I realized that https://github.com/bazelbuild/bazel/commit/78b8871c6b9f93a8f5e4fcf3f32b1a2c8e1532c3 has not made the cut into Bazel 8 - was that intentional?

cc @Wyverald

luispadron commented 4 days ago

Do you still have the file /Users/build/.jenkins/workspace/cash-ios/ios-builder/s/c/hs_err_pid55320.log?

Unfortunately we don't collect these right now but I'm going to start doing that soon and report back once we get a log to share.

fmeum commented 4 days ago

While looking at this, I realized that 78b8871 has not made the cut into Bazel 8 - was that intentional?

It wasn't, but if this issue really only shows up in 7.4, then that commit may actually be what caused it. I haven't checked whether the JDK can be updated further.

ivan-golub commented 4 days ago

On our side we see consistent stream of SIGBUS/SIGSEGV on 7.2 running on Macs.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x000000041914bb80, pid=1927, tid=49963
#
# JRE version: OpenJDK Runtime Environment Zulu21.28+85-CA (21.0+35) (build 21+35)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.28+85-CA (21+35, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x000000041914bb80
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib:/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib/server:/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/ -Dfile.encoding=ISO-8859-1 -Duser.country= -Duser.language= -Duser.variant= -DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 -Xmx30g /Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/A-server.jar --max_idle_secs=10800 --shutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64 --install_base=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c --install_md5=f4e81c77e3f92a1e901b18f8adecf60c --output_base=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f --workspace_directory=/private/var/lib/jenkins/workspace/phantom --default_system_javabase=/nix/store/l57is3k03hrl5ds9vmnnzmvy1izx9fw7-zulu17.44.53-ca-jdk-17.0.8.1 --failure_detail_out=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 --host_jvm_args=-Xmx30g --product_name=Bazel --option_sources=home_Urc::host_Ujvm_Uargs:/private/var/lib/jenkins/workspace/phantom/.bazelrc:output_Uuser_Uroot:/Users/devicelab/Snapchat/Dev/.cache/bzl/default.451.bazelrc:shutdown_Uon_Ulow_Usys_Umem:/Users/devicelab/Snapchat/Dev/.cache/bzl/default.451.bazelrc:system_Urc::workspace_Urc:

Host: "Mac13,2" arm64, 20 cores, 128G, Darwin 23.5.0, macOS 14.5 (23F79)
Time: Wed Nov 20 01:24:47 2024 GMT elapsed time: 44.782824 seconds (0d 0h 0m 44s)

---------------  T H R E A D  ---------------

Current thread (0x000000012b8f9600):  JavaThread "skyframe-evaluator-cpu-heavy-18" daemon [_thread_in_Java, id=49963, stack(0x0000000b0d064000,0x0000000b0d267000) (2060K)]

Stack: [0x0000000b0d064000,0x0000000b0d267000],  sp=0x0000000b0d265ab0,  free space=2054k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x000000041914bb80
j  java.lang.invoke.LambdaForm$MH+0x000000700075c800.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+59 java.base@21
J 14847 c2 net.starlark.java.eval.MethodDescriptor.call(Ljava/lang/Object;[Ljava/lang/Object;Lnet/starlark/java/eval/Mutability;)Ljava/lang/Object; (373 bytes) @ 0x0000000111bc90cc [0x0000000111bc8f40+0x000000000000018c]
J 16212 c2 net.starlark.java.eval.BuiltinFunction.fastcall(Lnet/starlark/java/eval/StarlarkThread;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (53 bytes) @ 0x00000001113f46c0 [0x00000001113f4600+0x00000000000000c0]
J 16297 c2 net.starlark.java.eval.Starlark.fastcall(Lnet/starlark/java/eval/StarlarkThread;Ljava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (143 bytes) @ 0x00000001114f921c [0x00000001114f9180+0x000000000000009c]
J 18557 c2 net.starlark.java.eval.Eval.evalCall(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/CallExpression;)Ljava/lang/Object; (579 bytes) @ 0x0000000111ea3838 [0x0000000111ea2a00+0x0000000000000e38]
J 18570 c2 net.starlark.java.eval.Eval$1Lambda.execClauses(I)V (291 bytes) @ 0x0000000111e87428 [0x0000000111e85080+0x00000000000023a8]
J 11383 c2 net.starlark.java.eval.Eval.evalComprehension(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/Comprehension;)Ljava/lang/Object; (72 bytes) @ 0x00000001116bd1d4 [0x00000001116bcec0+0x0000000000000314]
J 18557 c2 net.starlark.java.eval.Eval.evalCall(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/CallExpression;)Ljava/lang/Object; (579 bytes) @ 0x0000000111ea73b8 [0x0000000111ea2a00+0x00000000000049b8]
J 6397 c2 net.starlark.java.eval.Eval.eval(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/Expression;)Ljava/lang/Object; (322 bytes) @ 0x000000011101478c [0x00000001110145c0+0x00000000000001cc]
J 7846 c2 net.starlark.java.eval.Eval.execStatements(Lnet/starlark/java/eval/StarlarkThread$Frame;Ljava/util/List;Z)Lnet/starlark/java/syntax/TokenKind; (171 bytes) @ 0x0000000111201f80 [0x0000000111201d00+0x0000000000000280]
J 10349 c2 net.starlark.java.eval.Eval.execIf(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/IfStatement;)Lnet/starlark/java/syntax/TokenKind; (47 bytes) @ 0x00000001114eb350 [0x00000001114eb100+0x0000000000000250]
J 7846 c2 net.starlark.java.eval.Eval.execStatements(Lnet/starlark/java/eval/StarlarkThread$Frame;Ljava/util/List;Z)Lnet/starlark/java/syntax/TokenKind; (171 bytes) @ 0x000000011120213c [0x0000000111201d00+0x000000000000043c]
J 18543 c2 net.starlark.java.eval.StarlarkFunction.fastcall(Lnet/starlark/java/eval/StarlarkThread;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (127 bytes) @ 0x0000000111e388b4 [0x0000000111e38640+0x0000000000000274]
J 16297 c2 net.starlark.java.eval.Starlark.fastcall(Lnet/starlark/java/eval/StarlarkThread;Ljava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (143 bytes) @ 0x00000001114f921c [0x00000001114f9180+0x000000000000009c]
J 19980 c1 com.google.devtools.build.lib.analysis.starlark.StarlarkRuleConfiguredTargetUtil.evalRule(Lcom/google/devtools/build/lib/analysis/RuleContext;Lcom/google/devtools/build/lib/packages/RuleClass;)Ljava/lang/Object; (189 bytes) @ 0x000000010a4cf210 [0x000000010a4cef40+0x00000000000002d0]
J 17167 c1 com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(Lcom/google/devtools/build/lib/analysis/AnalysisEnvironment;Lcom/google/devtools/build/lib/packages/Rule;Lcom/google/devtools/build/lib/analysis/config/BuildConfigurationValue;Lcom/google/devtools/build/lib/skyframe/ConfiguredTargetKey;Lcom/google/devtools/build/lib/util/OrderedSetMultimap;Lcom/google/devtools/build/lib/analysis/config/ConfigConditions;Lcom/google/devtools/build/lib/analysis/ToolchainCollection;Lcom/google/devtools/build/lib/collect/nestedset/NestedSet;Lcom/google/devtools/build/lib/analysis/ExecGroupCollection$Builder;)Lcom/google/devtools/build/lib/analysis/ConfiguredTarget; (519 bytes) @ 0x000000010a6d6368 [0x000000010a6d3d40+0x0000000000002628]
J 18274 c2 com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(Lcom/google/devtools/build/lib/analysis/AnalysisEnvironment;Lcom/google/devtools/build/lib/actions/ArtifactFactory;Lcom/google/devtools/build/lib/packages/Target;Lcom/google/devtools/build/lib/analysis/config/BuildConfigurationValue;Lcom/google/devtools/build/lib/skyframe/ConfiguredTargetKey;Lcom/google/devtools/build/lib/util/OrderedSetMultimap;Lcom/google/devtools/build/lib/analysis/config/ConfigConditions;Lcom/google/devtools/build/lib/analysis/ToolchainCollection;Lcom/google/devtools/build/lib/collect/nestedset/NestedSet;Lcom/google/devtools/build/lib/analysis/ExecGroupCollection$Builder;)Lcom/google/devtools/build/lib/analysis/ConfiguredTarget; (414 bytes) @ 0x0000000111e55798 [0x0000000111e55700+0x0000000000000098]
J 18239 c2 com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(Lcom/google/devtools/build/lib/skyframe/SkyframeBuildView;Lcom/google/devtools/build/skyframe/SkyFunction$Environment;Lcom/google/devtools/build/lib/analysis/TargetAndConfiguration;Lcom/google/devtools/build/lib/skyframe/ConfiguredTargetKey;Lcom/google/devtools/build/lib/util/OrderedSetMultimap;Lcom/google/devtools/build/lib/analysis/config/ConfigConditions;Lcom/google/devtools/build/lib/analysis/ToolchainCollection;Lcom/google/devtools/build/lib/analysis/ExecGroupCollection$Builder;Lcom/google/devtools/build/lib/collect/nestedset/NestedSet;)Lcom/google/devtools/build/lib/analysis/ConfiguredTargetValue; (395 bytes) @ 0x0000000111e0486c [0x0000000111e04180+0x00000000000006ec]
j  com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(Lcom/google/devtools/build/skyframe/SkyKey;Lcom/google/devtools/build/skyframe/SkyFunction$Environment;)Lcom/google/devtools/build/skyframe/SkyValue;+414
J 16324 c2 com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run()V (1927 bytes) @ 0x00000001116f89bc [0x00000001116f7b40+0x0000000000000e7c]

or 

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000001064ae91c, pid=46540, tid=14083
#
# JRE version: OpenJDK Runtime Environment Zulu21.28+85-CA (21.0+35) (build 21+35)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.28+85-CA (21+35, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0x3f291c]  G1ConcurrentMark::mark_in_bitmap(unsigned int, oopDesc*)+0x3c
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/rules_xcodeproj.noindex/build_output_base/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib:/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib/server:/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/ -Dfile.encoding=ISO-8859-1 -Duser.country= -Duser.language= -Duser.variant= -DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 -Xmx30g /Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/A-server.jar --max_idle_secs=10800 --shutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/Users/ci/.cache/bazel/arm64 --install_base=/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c --install_md5=f4e81c77e3f92a1e901b18f8adecf60c --output_base=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/rules_xcodeproj.noindex/build_output_base --workspace_directory=/private/var/lib/jenkins/workspace/phantom --default_system_javabase=/Library/Java/JavaVirtualMachines/temurin-23.jdk/Contents/Home --failure_detail_out=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/rules_xcodeproj.noindex/build_output_base/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 --host_jvm_args=-Xmx30g --host_jvm_args=-Xdock:name=/Applications/Xcode16.0_16A242d.app/Contents/Developer --product_name=Bazel --option_sources=home_Urc::host_Ujvm_Uargs::output_Ubase::output_Uuser_Uroot:/Users/ci/.cache/bzl/default.451.bazelrc:shutdown_Uon_Ulow_Usys_Umem:/Users/ci/.cache/bzl/default.451.bazelrc:system_Urc::workspace_Urc:

Host: "Mac13,2" arm64, 20 cores, 128G, Darwin 23.5.0, macOS 14.5 (23F79)
Time: Wed Nov 20 08:28:59 2024 GMT elapsed time: 869.441221 seconds (0d 0h 14m 29s)

---------------  T H R E A D  ---------------

Current thread (0x0000000135a04190):  WorkerThread "G1 Conc#0"      [id=14083, stack(0x000000016baa4000,0x000000016bca7000) (2060K)]

Stack: [0x000000016baa4000,0x000000016bca7000],  sp=0x000000016bca4c30,  free space=2051k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0x3f291c]  G1ConcurrentMark::mark_in_bitmap(unsigned int, oopDesc*)+0x3c
V  [libjvm.dylib+0x3d8fc4]  G1CMTask::make_reference_grey(oopDesc*)+0x24
V  [libjvm.dylib+0x3f1da4]  void OopOopIterateDispatch<G1CMOopClosure>::Table::oop_oop_iterate<InstanceRefKlass, narrowOop>(G1CMOopClosure*, oopDesc*, Klass*)+0xe4
V  [libjvm.dylib+0x3ee6f4]  void G1CMTask::process_grey_task_entry<true>(G1TaskQueueEntry)+0x224
V  [libjvm.dylib+0x3e72f8]  G1CMTask::drain_local_queue(bool)+0x150
V  [libjvm.dylib+0x3ecf30]  G1CMTask::do_marking_step(double, bool, bool)+0x16c
V  [libjvm.dylib+0x3eeb4c]  G1CMConcurrentMarkingTask::work(unsigned int)+0xa8
V  [libjvm.dylib+0xae482c]  WorkerThread::run()+0x94
V  [libjvm.dylib+0xa28ef8]  Thread::call_run()+0xc8
V  [libjvm.dylib+0x865220]  thread_native_entry(Thread*)+0x158
C  [libsystem_pthread.dylib+0x6f94]  _pthread_start+0x88

siginfo: si_signo: 10 (SIGBUS), si_code: 1 (BUS_ADRALN), si_addr: 0x0000000936818df8
fmeum commented 3 days ago

Cc @cushon

cushon commented 3 days ago

I would try using the latest JDK update release and see if it still reproduces. If it does, and there's a repro, it would be good to report to https://bugreport.java.com/bugreport/crash.jsp

fmeum commented 3 days ago

@bazel-io fork 8.0.0

ivan-golub commented 3 days ago

Tried to update to java to zulu 21.38.21 in our fork (on top of 7.2.1) Seeing similar errors

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000004d5eccef8, pid=61097, tid=195087
#
# JRE version: OpenJDK Runtime Environment Zulu21.28+85-CA (21.0+35) (build 21+35)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.28+85-CA (21+35, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x00000004d5eccef8
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/embedded_tools/jdk/lib:/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/embedded_tools/jdk/lib/server:/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/ -Dfile.encoding=ISO-8859-1 -Duser.country= -Duser.language= -Duser.variant= -DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 -Xmx30g /Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/A-server.jar --max_idle_secs=10800 --shutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/Users/ci/.cache/bazel/arm64 --install_base=/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9 --install_md5=f188b2f86baffd6f7652f35ca90c6ae9 --output_base=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f --workspace_directory=/private/var/lib/jenkins/workspace/phantom --default_system_javabase=/nix/store/l57is3k03hrl5ds9vmnnzmvy1izx9fw7-zulu17.44.53-ca-jdk-17.0.8.1 --failure_detail_out=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 --host_jvm_args=-Xmx30g --product_name=Bazel --option_sources=home_Urc::host_Ujvm_Uargs:/private/var/lib/jenkins/workspace/phantom/.bazelrc:output_Uuser_Uroot:/Users/ci/.cache/bzl/default.451.bazelrc:shutdown_Uon_Ulow_Usys_Umem:/Users/ci/.cache/bzl/default.451.bazelrc:system_Urc::workspace_Urc:

Host: "Mac13,2" arm64, 20 cores, 128G, Darwin 23.5.0, macOS 14.5 (23F79)
Time: Thu Nov 21 14:13:44 2024 GMT elapsed time: 49.220094 seconds (0d 0h 0m 49s)

---------------  T H R E A D  ---------------

Current thread (0x0000000b260b5000):  JavaThread "skyframe-evaluator-cpu-heavy-4" daemon [_thread_in_Java, id=195087, stack(0x0000000aff064000,0x0000000aff267000) (2060K)]
Wyverald commented 3 days ago

While looking at this, I realized that 78b8871 has not made the cut into Bazel 8 - was that intentional?

It was actually cherry-picked as https://github.com/bazelbuild/bazel/commit/7a9e1ded2b8c382a05ead9f481e4e4755682f242 (on 11 Oct). If you're just looking at the GitHub UI, it doesn't tell you about cherry-picks, only whether the commit is actually on any branches.