Remote Build Execution upload batching features

[benbrittain]

• Queries the RBE Capabilities service to get maximum message size.
• Batches small blob updates into several BatchUploadBlobs requests.
• If file is larger than maximum message size, use Bytestream service for upload.

[benbrittain]

This is a subset of the improvements made in #181 but includes batching of smaller messages into a reduced number of BatchUpdateBlobs requests instead of exclusively using on ByteStream.Write and querying the max allowed size from the remote server.

It also includes downloading large file support in addition to uploading.

[benbrittain]

I tried running this on my "production" codebase at work as a lazy E2e test and the size of the protobuf messages is not quite right since the blob hashes aren't the only thing in a message. Putting together a fix in a minute.

There was a chance that batched messages would be sightly over the gRPC limit.

[benbrittain]

Fixed and rebased on main.

[benbrittain]

I've also added support for not uploading blobs already present on the remote

[benbrittain]

I'm get to fix the missed large blobs, but it's on my TODO list. I'd also like to make it so if for some reason a request fails in the batch it gets retried with the bytestream service

Sorry for the weird close/open. I'm definitely gonna finish this. Do you want these commits squashed into logical commits or fixes in subsequent commits?

[benbrittain]

This is ready for another look whenever you have time :)

[facebook-github-bot]

@krallin has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[benbrittain]

I have not forgotten about this, I'll add some upload testing soon :+1:

[krallin]

No problem, thanks! :)

On Fri, 28 Apr 2023 at 20:12, Benjamin Brittain @.***> wrote:

I have not forgotten about this, I'll add some upload testing soon 👍

— Reply to this email directly, view it on GitHub https://github.com/facebook/buck2/pull/193#issuecomment-1527923169, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANIHVQVHTPHVBJDX4KGEILXDQCCPANCNFSM6AAAAAAXIWD3TI . You are receiving this because you were mentioned.Message ID: @.***>

[benbrittain]

Added a few upload test cases (which, of course, caught a couple of minor problems)!

[facebook-github-bot]

@krallin has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[krallin]

Merged this today (after making a certain number of adjustements), thanks for your contribution to Buck2!