Closed avdv closed 3 months ago
So I don't know if this is the cause of this specific error, but symlinks in sources are basically completely unsupported. They sometimes kind of work, but you're in basically untested ground here so I'm not surprised that something has broken.
We do support symlinks in outputs, depending on what exactly it is you're doing it's possible that that may offer a path to working around this limitation
Thanks for your quick response!
So I don't know if this is the cause of this specific error, but buck2 doesn't support symlinks in sources are basically completely unsupported. They sometimes kind of work, but you're in basically untested ground here so I'm not surprised that something has broken.
OK, fair enough. In this case it seems to indicate a problem with handling the http2 responses gracefully. I would guess that the upstream server "sees" that some files are already uploaded and replies with cancelling the stream, which just should be ignored perhaps?!
We do support symlinks in outputs, depending on what exactly it is you're doing it's possible that that may offer a path to working around this limitation
The same problem turns up when we use the output of yarn install
as input to another action running remotely. My current workaround is to create a tarball inside a local action, and then explicitly unpack the tarball before doing the real work of the action.
The same problem turns up when we use the output of yarn install as input to another action running remotely. My current workaround is to create a tarball inside a local action, and then explicitly unpack the tarball before doing the real work of the action.
Oof, yeah this sounds very likely to be a bug. This probably requires figuring out exactly what the buck2-RE communication looks like and which of the two is out of spec (guessing that it's us is a good default). I think we probably have logging that can help with that?
I turned up logging for grpc calls in the bazel-remote-worker (it's explicitly disabled in code in order to avoid "Received DATA frame for an unknown stream 1521" error messages) and got this:
240229 08:43:39.770:WT 17 [io.grpc.netty.NettyServerStream$TransportState.deframeFailed] Exception processing message
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 4594008
at io.grpc.Status.asRuntimeException(Status.java:526)
at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:391)
at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:271)
at io.grpc.internal.MessageDeframer.request(MessageDeframer.java:161)
at io.grpc.internal.AbstractStream$TransportState$1RequestRunnable.run(AbstractStream.java:236)
at io.grpc.netty.NettyServerStream$TransportState$1.run(NettyServerStream.java:202)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
The maximum batch size is set to 4 1000 1000. I don't know why the message is so much larger than that limit. Maybe it's because symlinks are involved?
The maximum batch size is set to 4 1000 1000. I don't know why the message is so much larger than that limit. Maybe it's because symlinks are involved?
Oh, it's probably just because of the overhead. For each datum, there are at least 72 extra Bytes needed to transmit the hash and the compressor enum value. For one example request that I looked at, there were 7410 entries in one batch; which already adds up to 533520 Bytes. Plus a few Bytes needed for encoding every element of the requests
field.
We have increased the max inbound message size for the bazel-remote-worker to an arbitrarily high number in order to workaround that issue and have not seen this error again.
Closing, since I think nothing is to be done here.
For the following
BUCK
file:when using remote execution (we are currently using the bazel-remote-worker locally) we see the following error:
This seems to be related to the structure of the srcs of the filegroup since the
public
folder contains symlinks into thesrc
folder:After removing either
public/icons
orpublic/webfonts
, the upload succeeds. And once it succeeded, it also succeeds when the symlinks are restored:Also, I noticed that the symlinks are not preserved in the
buck-out/v2/gen/root/524f8da68ea2a374/frontend/__assest__
directory:Is this to be expected?
BTW, I am using
buck2 aa5cc9e36218b3afcad06608d91f9e8baa1d5c88e0b2a2f561b1b695a320afc7
, the 2024-01-02 pre-release.cc: @aherrmann