search

buchgr / bazel-remote

A remote cache for Bazel
https://bazel.build
Apache License 2.0
595 stars 154 forks source link

BAZEL_REMOTE_ENABLE_AC_KEY_INSTANCE_MANGLING env makes the service point to /data #669

Open marcelodeaguiar opened 1 year ago

marcelodeaguiar commented 1 year ago

When passing the parameter BAZEL_REMOTE_ENABLE_AC_KEY_INSTANCE_MANGLING as true when using Kubernets and passing the BAZEL_REMOTE_DIR env witha value like /cacheDir the service is pointing always to /data instead.

I've noted the comments on the README about mangling but did not understand exactly what is the impact on this. Should I have passes cacheDir instead? (whithout the /)

Disabling mangling solves the problem.

mostynb commented 1 year ago

This is unexpected. When the enable_ac_key_instance_mangling feature is enabled, bazel-remote is supposed to take the action cache hash and hash that along with the instance string, to give a new internal action cache lookup key, so different instances effectively have separate action caches. It should not change the data dir at all.

The docker container uses the path /data internally, when running with the docker runtime you're supposed to map a host directory to /data inside the container. I wonder if there's a kubernetes misconfiguration happening here...

Are you able to reproduce this issue without kubernetes? (That would be a lot easier for me to debug.)

marcelodeaguiar commented 1 year ago

I had no time to try it on the docker runtime. As soon as I can I will test on that and let you know.

mostynb commented 1 year ago

Something to double check with kubernetes deployments- if it is named bazel-remote or something similar, then some environment variables will be set that can conflict with your configuration. This has tripped up a several people before.