Cache restore not working - connect ETIMEDOUT 10.0.0.8:9000

[enricojonas]

Hi,

I am trying to use the cache action with self-hosted runners (in GKE) towards a GHES instance. It seems that saving the cache is just fine, but on restore I receive error connect ETIMEDOUT 10.0.0.8:9000

Environment: GHES 3.7.4 Self Hosted runner in GKE (base image Ubuntu 22.04 / Runner Agent 2.298.2)

Saving cache:

Post job cleanup.
/usr/bin/tar --posix -cf cache.tgz --exclude cache.tgz -P -C /runner/_work/repo/repo --files-from manifest.txt -z
Cache Size: ~436 MB (456671309 B)
Cache saved successfully
Cache saved with key: Linux-maven-b478786cd53f1de0795ba40ef14f77c81b6ea366bb1970e27f680154fee686c5

Restoring cache:

 ##[debug]Resolved Keys:
##[debug]["Linux-maven-b478786cd53f1de0795ba40ef14f77c81b6ea366bb1970e27f680154fee686c5","Linux-maven-"]
##[debug]Checking zstd --version
##[debug]Unable to locate executable file: zstd. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
##[debug]
##[debug]Resource Url: https://<git-server-url>/_services/artifactcache/aysbnhh9gKLGnX4JIYuIQjZECZmeQnOBpj8GsGJwFdlfESQdA4/_apis/artifactcache/cache?keys=Linux-maven-b478786cd53f1de0795ba40ef14f77c81b6ea366bb1970e27f680154fee686c5%2CLinux-maven-&version=c538a492dc53daaed119aaacbe9be444dd6f0478f3c1e96eef74be5e3b7be6fa
::add-mask::***
##[debug]Cache Result:
##[debug]***"scope":"refs/pull/2/merge","cacheKey":"Linux-maven-b478786cd53f1de0795ba40ef14f77c81b6ea366bb1970e27f680154fee686c5","cacheVersion":"c538a492dc53daaed119aaacbe9be444dd6f0478f3c1e96eef74be5e3b7be6fa","creationTime":"2023-02-08T17:43:30.33Z","archiveLocation":"***"***
##[debug]Archive Path: /runner/_work/_temp/6219a271-8daf-4924-8d78-8d1f70ac401a/cache.tgz
##[debug]Use Azure SDK: true
##[debug]Download concurrency: 8
##[debug]Request timeout (ms): 30000
##[debug]Cache segment download timeout mins env var: undefined
##[debug]Segment download timeout (ms): 3600000
##[debug]downloadCache - Attempt 1 of 2 failed with error: connect ETIMEDOUT 10.0.0.8:9000
##[debug]downloadCache - Attempt 2 of 2 failed with error: connect ETIMEDOUT 10.0.0.8:9000
Warning: Failed to restore: downloadCache failed: connect ETIMEDOUT 10.0.0.8:9000
Cache not found for input keys: Linux-maven-b478786cd53f1de0795ba40ef14f77c81b6ea366bb1970e27f680154fee686c5, Linux-maven-
##[debug]Node Action run completed with exit code 0
##[debug]MSYS='winsymlinks:nativestrict'
##[debug]Save intra-action state CACHE_KEY = Linux-maven-b478786cd53f1de0795ba40ef14f77c81b6ea366bb1970e27f680154fee686c5
##[debug]Finishing: Cache local Maven repository

I cannot figure out what is wrong here.

Thanks

[hendrik-schaffer]

Hi @enricojonas ,

From the debug output

[debug]Checking zstd --version

[debug]Unable to locate executable file: zstd. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

I think we have been run into a similar issue in the past and the solution was to simply provide this binary and add it to the PATH on the self-hosted runner.

[enricojonas]

@hendrik-schaffer thanks for reaching out. I have tried to integrate this and it didn't do any change. It seems to be connected to how our GHES is setup and it's using the storage backend through that address which is not reachable from the runners - so the error message is not very descriptive. I will close this issue here as we this is an infrastructure problem.

[lvpx]

@enricojonas I would suggest creating a Support ticket for this since it is related to GHES.

[enricojonas]

@pdotl yes, that's what we did. Github support helped us to identify the issue.

[brignano]

@pdotl yes, that's what we did. Github support helped us to identify the issue.

What was the issue?

[enricojonas]

The issue was that we were using MinIO storage - apparently it is not working with this.

[brignano]

The issue was that we were using MinIO storage - apparently it is not working with this.

Thanks for the quick reply! I've been trying to resolve the same problem with GHES but haven't found a solution yet.

[thexperiments]

Same problem here, did you find any solution. However we are using DELL ISILON S3 compatible storage