Closed brentleyjones closed 11 months ago
According to the stack trace:
java.io.IOException: /private/var/tmp/_bazel_iosci/7c81b0540945e51fddc7bd01ee8d0abb/execroot/lyftios/bazel-out/darwin-fastbuild/bin/tools/CodeAnalysis/Tests/LinuxMain.swift.tmp (Not a directory)
| at com.google.devtools.build.lib.unix.NativePosixFiles.openWrite(Native Method)
| at com.google.devtools.build.lib.unix.UnixFileSystem.createFileOutputStream(UnixFileSystem.java:493)
| at com.google.devtools.build.lib.vfs.AbstractFileSystem.getOutputStream(AbstractFileSystem.java:174)
| at com.google.devtools.build.lib.vfs.AbstractFileSystem.getOutputStream(AbstractFileSystem.java:188)
| at com.google.devtools.build.lib.vfs.Path.getOutputStream(Path.java:425)
| at com.google.devtools.build.lib.vfs.Path.getOutputStream(Path.java:413)
| at com.google.devtools.build.lib.remote.common.LazyFileOutputStream.ensureOpen(LazyFileOutputStream.java:66)
| at com.google.devtools.build.lib.remote.common.LazyFileOutputStream.write(LazyFileOutputStream.java:42)
| at com.google.devtools.build.lib.remote.RemoteCache$ReportingOutputStream.write(RemoteCache.java:510)
| at com.google.devtools.build.lib.remote.util.DigestOutputStream.write(DigestOutputStream.java:58)
| at java.base/java.io.FilterOutputStream.write(Unknown Source)
| at com.google.protobuf.ByteString$LiteralByteString.writeTo(ByteString.java:1381)
| at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onNext(GrpcCacheClient.java:352)
| ... 13 more
it calls into NativePosixFiles.openWrite() which calls open
which throws "Not a directory" or ENOTDIR
.
According to the doc, ENOTDIR
means "A component of the path prefix is not a directory". So I am guessing the issue is the output directory is not prepared probably for the action.
Might related https://github.com/bazelbuild/bazel/issues/6393.
@brentleyjones Is that possible to create a repro?
It's weird, the parent directories are actually created before the download.
It was on CI. All I can think of is the directories failed to be created, because of low disk space.
We're seeing what I think is the same issue with bazel 5.1.1:
[1,147 / 1,149] Compiling test/extensions/network/dns_resolver/apple/apple_dns_impl_test.cc; 1s remote-cache
ERROR: /Users/runner/work/1/s/test/extensions/network/dns_resolver/apple/BUILD:11:14: Compiling test/extensions/network/dns_resolver/apple/apple_dns_impl_test.cc failed: Exec failed due to IOException: 41 errors during bulk transfer:
java.io.IOException: io.grpc.StatusRuntimeException: CANCELLED: Failed to read message.
java.io.IOException: io.grpc.StatusRuntimeException: CANCELLED: Failed to read message.
Unfortunately I think we're hiding the full stack trace https://dev.azure.com/cncf/envoy/_build/results?buildId=106443&view=logs&jobId=fa3d3e18-6969-5713-c3e7-3581195704fd&j=fa3d3e18-6969-5713-c3e7-3581195704fd&t=ba370bef-ca49-52f5-0c93-7c0c0f27c465 (this log will probably rotate)
Hit this bug today when trying to use
build --strategy=GoLink=linux-sandbox,remote
build --strategy=GoCompile=linux-sandbox,remote
so that I could do GoLinking locally while doing the rest of my build on RBE.
Not sure if that information is helpful
A lot of things have changed since 5. Please reopen if it's still happening.
Description of the problem / feature request:
We encountered an error where bazel fails to read from the remote cache because it throws an execution locally. Seems to be from this code path: https://github.com/bazelbuild/bazel/commit/cff2ea55dafc57d7d2db813522942e2ee137ed08#diff-c0c72bba110eca25447686392b1f265feabe77c8a53ed7387f70f82bcdaba441R112.
LinuxMain.swift
is single file output from agenrule
.Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
I don't know of one.
What operating system are you running Bazel on?
macOS 11.6.1
What's the output of
bazel info release
?release 5.0.0rc1