runner wont use cache server although they can communicate

[phk-nord]

This kind of relates to #66 but the cause seems to be different as both services can communicate with another.

About the setup I have the cache server running with the helm chart thats open as PR #58 and the runner as ARC ephemeral Runner Scale set following the official documentation.

Following your suggestion from #66 zstd is installed on the runner image as well as the sed command is executed during the docker build of the runner image.

my runner can curl the cache server by resolving the k8s service

runner@gh-runner-nordeck-2497n-runner-js4h6:~$ curl http://gh-cache-github-actions-cache-server.gh-runner.svc.cluster.local
OK

The shortened cache server logs (unnecessary Get / requests and same error arrays removed) indicate that communication in both directions seem possible

[cache-server] ℹ Cleaning up cache entries older than 30d with schedule 0 0 * * * (next run: 10/25/2024, 12:00:00 AM)
[cache-server] ℹ 🚀 Starting GitHub Actions Cache Server (v3.1.0)
[cache-server] ℹ Using database driver: sqlite
[cache-server] ℹ Migrating database...
[cache-server] ⚙ Migration results [ { migrationName: '2024-04-20T11:18:44', direction: 'Up', status: 'Success' } ]
[cache-server] ✔ Database migrated
[cache-server] ℹ Using storage driver: filesystem
Listening on http://[::]:3000
[cache-server] ⚙ Request: GET /
[cache-server] ⚙ Response: GET / > 200
[cache-server] ⚙ Request: GET /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache?keys=trivy-binary-v0.56.1-Linux-X64&version=460b6211f9316394cdbffb7a337e3c470e74c8eedbb91394e2b22d96db894c90

[cache-server]  ERROR  Response: GET /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache?keys=trivy-binary-v0.56.1-Linux-X64&version=460b6211f9316394cdbffb7a337e3c470e74c8eedbb91394e2b22d96db894c90 > 404
 Cannot find any route matching /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache.

  at createError$1 (server/chunks/runtime.mjs:1886:15)
  at matchHandler (server/chunks/runtime.mjs:3017:16)
  at Object.handler (server/chunks/runtime.mjs:3056:19)
  at Object.handler (server/chunks/runtime.mjs:2832:31)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
  at async Server.toNodeHandle (server/chunks/runtime.mjs:3102:7)

[cache-server] ⚙ Request: GET /
[cache-server] ⚙ Response: GET / > 200
[cache-server] ⚙ Request: GET /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache?keys=cache-trivy-2024-10-24%2Ccache-trivy-&version=a9b2fa181c65f144908fd4eb9469645f2282e067b1beb3c071e9d4a984b8caae

[cache-server]  ERROR  Response: GET /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache?keys=cache-trivy-2024-10-24%2Ccache-trivy-&version=a9b2fa181c65f144908fd4eb9469645f2282e067b1beb3c071e9d4a984b8caae > 404
 Cannot find any route matching /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache.

[cache-server] ⚙ Request: GET /
[cache-server] ⚙ Response: GET / > 200
[cache-server] ⚙ Request: POST /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches

[cache-server]  ERROR  Response: POST /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches > 404
 Cannot find any route matching /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches.

[cache-server] ⚙ Request: GET /
[cache-server] ⚙ Response: GET / > 200
[cache-server] ⚙ Request: POST /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches

[cache-server]  ERROR  Response: POST /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches > 404
 Cannot find any route matching /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches.

[cache-server] ⚙ Request: POST /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches
[cache-server] ⚙ Request: POST /6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/caches
[cache-server] ⚙ Request: GET /
[cache-server] ⚙ Response: GET / > 200

I tested the caching on the same action that build our runner image and generates an sbom with trivy. The following is are the most relevant snippets of the trivy step (in action debug mode) as that uses caching usually.

Somehow the runner tries to cache it locally but never uses that directory anyways

2024-10-25T07:50:26.3015208Z ##[debug]..=> 'cache-dir'
2024-10-25T07:50:26.3017615Z ##[debug]=> '/home/runner/_work/gh-runner-image/gh-runner-image/.cache/trivy'
2024-10-25T07:50:26.3020935Z ##[debug]Result: '/home/runner/_work/gh-runner-image/gh-runner-image/.cache/trivy'

2024-10-25T07:50:26.3232791Z ##[endgroup]
2024-10-25T07:50:26.5809362Z ##[debug]Resolved Keys:
2024-10-25T07:50:26.5826421Z ##[debug]["cache-trivy-2024-10-25","cache-trivy-"]
2024-10-25T07:50:26.5852799Z ##[debug]Checking zstd --quiet --version
2024-10-25T07:50:26.6198389Z ##[debug]1.4.8
2024-10-25T07:50:26.6222650Z ##[debug]zstd version: 1.4.8
2024-10-25T07:50:26.6233448Z ##[debug]Resource Url: http://gh-cache-github-actions-cache-server.gh-runner.svc.cluster.local/6PoP3tcGBCllhkzh2JByVG4XZZ9z3K3Z_apis/artifactcache/cache?keys=cache-trivy-2024-10-25%2Ccache-trivy-&version=da177c2d253528ed325846debfddbf7f46bb97cf1e6eb23b3614f557634da93a
2024-10-25T07:50:26.6749184Z ##[warning]Failed to restore: Cache service responded with 404
2024-10-25T07:50:26.6764170Z ##[debug]Failed to delete archive: Error: ENOENT: no such file or directory, unlink ''
2024-10-25T07:50:26.6769847Z Cache not found for input keys: cache-trivy-2024-10-25, cache-trivy-
2024-10-25T07:50:26.6833475Z ##[debug]Node Action run completed with exit code 0
2024-10-25T07:50:26.6923718Z ##[debug]Save intra-action state CACHE_KEY = cache-trivy-2024-10-25
2024-10-25T07:50:26.6958172Z ##[debug]Finished: run

I am happy if you can give me any pointers or ideas that could prevent communication or use of the caching

[LouisHaftmann]

Does your ACTIONS_CACHE_URL have a trailing slash?

[phk-nord]

it doesn't no it's http://gh-cache-github-actions-cache-server.gh-runner.svc.cluster.local/TOKEN

[LouisHaftmann]

Try adding a trailing slash

[phk-nord]

adding the trailing / to the ACTIONS_CACHE_URL did solve all 404 errors the runner reported. Caches can be created and retrieved successfully. Thank you for your help.