Binary patching alternative

[caquino]

Hi, I'm evaluating some local caching options for self-hosted runners and ended up here, one requirement that worries me a little bit is the requirement to patch the runner binary to allow the environment variable to be set to what is required.

As far as I understand the ACTIONS_CACHE_URL return different values based in your geolocation, like for example https://acghubeus1.actions.githubusercontent.com/ would it be possible instead of patching to provide within the runners the either of the following solutions:

Option 1 hijack the domain *.actions.githubusercontent.com

• dnsmasq that answers authoritative for *.actions.githubusercontent.com and sends it to the cache server
• ca root certificates that allows to self-sign for githubusercontent.com

possible issues: Is anything else other than cache using this domains? can it break something else

Options 2 abuse the proxy support on the runner to hijack the cache functionality

• configure the proxy environment variables pointing to a reverse proxy (e.g. nginx)
• configured nginx to intercep caching requests and forward to the custom cache server

Option 3 remap system fuctions using LD_PRELOAD

• reverse enginer the code to see what functionality is used to write to cache, e.g: write edit actually we can remap the getenv function and when it attempts to read ACTIONS_CACHE_URL we can send another response, this may be a good solution
• use LD_PRELOAD to map to a custom function that intercepts and sends to the custom cache (similar to this: https://github.com/caquino/ldsniff)

I haven't tested any of the options as I don't have a lab available at the moment but I will try to put a POC in place when I get some spare time and report back.

Thanks!

EDIT

I can confirm the LD_PRELOAD approach works you can point to HTTP or https endpoints and it will respect the protocol:

As you can see here I pointed to a local HTTP server and it was able to identify the 404 error.

and I could also confirm in the logs:

127.0.0.1 - - [08/Apr/2024:18:16:00 +0000] "GET /_apis/artifactcache/cache?keys=Linux-my-cache&version=8a1a35c329bc005ec3aa86e16d05dde409e80ca147480a4f22f31c778450b077 HTTP/1.1" 404 162 "-" "actions/cache"

You can check the source code I used here: https://github.com/caquino/runner/tree/main/envinject

Obviously this code can be improved to allow configuration, or even better to allow the override of ANY environment by adding a prefix OVERRIDE_* which would make it a more generic tool.

I did the bare minimum to test it locally, sorry for the dumb code.

To start the self-hosted runner I used the following command: LD_PRELOAD=/path/to/envinject/envinject.so ./run.sh

I also tested overriding the environment variable to a local version of this repo running on docker which worked successfully.

[LouisHaftmann]

Thanks for opening this issue :heart:

Patching the binary is a bit of a hacky solution and I appreciate the time you took to try out other solutions.

I believe the proxy approach would be the easiest to set up, as it only requires some env vars and no files or additional binaries.

We'll definitely look into it!

[caquino]

I would say that you listed pretty much the pros for the proxy approach. If we try to list the cons we have:

• No granularity. Now your proxy needs to handle 100% of the traffic
• Proxy needs to handle CONNECT method (tested and most requests are using CONNECT while proxied)

Both other options have more granularity, but you also have downsides on option 1:

• Requires you to setup a much more supporting infra (dns/CA)
• As granular as DNS, if the same domain is used for multiple things it will require specialized code/configuration

For option 3:

• Requires different approach depending on the OS (Linux/BSD/OSX works with little change, Windows will require a DLL as far as I understand it is possible)

[LouisHaftmann]

We'll try out the proxy approach. Adding proxy functionality shouldn't introduce breaking changes so you could still use option 3 on your setup.

[LouisHaftmann]

Unfortunately the proxy approach doesn't work. We cannot catch the action runner's requests because it uses https.

[joh-klein]

I just came across this and found this drop-in replacement for actions/cache – did you see that? They expose export CIRRUS_HTTP_CACHE_HOST=cache.internal:8080 which would then work with your server.