search

notnoopci / bazel-remote-proxy

MIT License
38 stars 5 forks source link

Circle CI backend fails executing Genrule #1

Open Steve-Munday opened 6 years ago

Steve-Munday commented 6 years ago

This remote proxy looks like exactly what I need to get Bazel Remote Cache setup on my Circle CI builds, but it's currently failing for me. I have it set up to install via curl according to your docs, and I have a build step create a .bazelrc with everything I need for Bazel in CircleCI:

- run:
          name: Configure Bazel environment
          command: |
            echo "build --local_resources=10240,6,1.0 --jobs=6" >> .bazelrc
            echo "build --spawn_strategy=remote --genrule_strategy=remote" >> .bazelrc
            echo "build --strategy=Javac=remote --strategy=Closure=remote" >> .bazelrc
            echo "build --remote_rest_cache=http://localhost:7654" >> .bazelrc

However my Bazel build step fails almost immediately upon executing a genrule:

ERROR: /home/circleci/<PATH>/BUILD:286:1: Executing genrule //libmuse:muse_bridge_h failed (Exit 34). Note: Remote connection/protocol failed with: execution failed

Any idea what might be going on? @notnoopci

buchgr commented 6 years ago

@alexeagle maybe?

buchgr commented 6 years ago

@Steve-Munday where do you start the caching backend? Also, are you sure your credentials for your AWS S3 bucket are working?

Steve-Munday commented 6 years ago

As per the docs, I tried to avoid using an S3 bucket by using the CircleCI cache backend:

"When running inside CircleCI 2.0 build environment, you can use CircleCI caching storage - eliminating the need to manage one's own S3 credentials and bucket."

Here is relevant Circle CI config:

       - run:
           name: Start Bazel remote proxy
           command: ~/bazel-remote-proxy -backend circleci://
           background: true

Maybe I can do some digging in Circle CI and see if any useful logs are generated.

alexeagle commented 6 years ago

I haven't had that problem - maybe compare with how we set up the remote cache for Angular? https://github.com/angular/angular/blob/master/docs/BAZEL.md#remote-cache

Steve-Munday commented 6 years ago

@alexeagle Thanks this was really helpful. I found a few differences between Angular's configuration and mine:

Now I'm seeing lots of requests in the remote proxy, but I am seeing this error in Bazel:

DEBUG: Upload to remote cache failed: PUT failed with status code 501

Is there something in your Circle project configuration that you did to enable this feature?

alexeagle commented 6 years ago

No, I don't think that angular/angular is special in any way. I listed all the config bits in that markdown file.

On Wed, Mar 14, 2018 at 2:24 PM Steve Munday notifications@github.com wrote:

@alexeagle https://github.com/alexeagle Thanks this was really helpful. I found a few things between Angular's configuration for this:

  • the docs in the README file here have the wrong port number. It should be 7643 not 7654. That was why my remote proxy failed to connect.
  • Angular appears to be using Bazel 0.9.0, with --experimental_remote_spawn_cache. I think this is no longer necessary as of 0.10.0? I put it in anyway.

Now I'm seeing this error:

DEBUG: Upload to remote cache failed: PUT failed with status code 501

Is there something in your Circle project configuration that you did to enable this feature?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/notnoopci/bazel-remote-proxy/issues/1#issuecomment-373179946, or mute the thread https://github.com/notifications/unsubscribe-auth/AAC5I7XEye3L2MM3PgP0zyq6o9rpSOzgks5teYqjgaJpZM4ST48r .

Steve-Munday commented 6 years ago

Thanks. Opening a support ticket with Circle at this point I think.

alexeagle commented 5 years ago

@Steve-Munday curious if you had any resolution for this?

Steve-Munday commented 5 years ago

Hi Alex. I was not able to get this working and couldn't spend any more time on it. Ultimately we decided not to use remote cache for our Bazel builds in Circle CI.

Instead, we use something more rudimentary, a local cache key for the bazel artifacts. We rotate the cache key to flush the cache monthly. It means our cache could be stale by up to a month, but even so, we still see a good enough performance improvement vs. not using a cache.

alisonbelow commented 4 years ago

@Steve-Munday what led you to conclude that the correct port is 7643 instead of 7654. I know it has been a while but digging around for how to remote cache bazel with CircleCI currently!