Open cloudshiftchris opened 1 month ago
As far as I can see, this is working as expected.
build-second
will start with a cache entry from build-first
is in the case that there's no cache entry from a previous build-second
run. The Gradle User Home cache entry then references all of the other entries (dependencies, wrapper-zips, etc)gradle-dependencies-v1-*
entry: this entry will be shared. But since the builds use different dependencies, they have separate dependency entries. Unfortunately if there is overlap between the sets of dependencies for different Jobs this results in some duplication of storage. (** see below)gradle-generated-gradle-jars-*
and gradle-wrapper-zips-*
entries are identical between the Jobs, so the Gradle User Home entries reference the exact same sub-entries: these sub-entries are shared between the jobs. So there is no duplication in the cache storage of this content.Note that when you re-run a Job, it doesn't get a new git SHA and thus you don't get a new cache key for Gradle User Home. This is why you see the "Entry not saved: referencing 'Gradle User Home' cache entry not saved". To test this properly, you'll need to push a dummy commit for each run.
Please check out https://github.com/gradle/actions/blob/main/docs/setup-gradle.md#how-gradle-user-home-caching-works for more details on how this works.
** There are definitely ways this could be made more efficient, if we either stored one big entry with all of the dependencies for all of the jobs, or by extracting the common dependencies into one entry and only storing the unique dependencies per-job.
Thanks for the clarification; have re-run tests (with dummy commits) - still seeing cleanups happening but presumably these relate to unused items within that build (e.g. removing the unused Groovy DSLs as its a Kotlin DSL script). Subsequent runs don't redownload artifacts, so good there.
Familiar with the documentation - what led me to believe that say dependencies were a shared cache across jobs:
1) The cache keys, as shown in the GHA cache summary, don't include job/step names, e.g. gradle-dependencies-v1-dadd467fc5b383227281c888d7a80dbc
vs gradle-home-v1|Linux-X64|build-second[27df40b80d25f4e0c5be504ed712742e]-dd00fdf2b4588b9de17b10ade0eb6d7891e723b3
, indicating (incorrectly) that these are separate from the job. (it isn't obvious that the opaque hash key represents the set of deps)
2) The documentation has a whole section on cache deduplication that lists certain caches as being "independent" which could be interpreted as "shared across jobs" (the bottom sentence there reinforces that, and indeed some caches are shared across jobs).
If these aren't indeed "shared" caches the documentation should be clearer on that - perhaps a table indicating what items are shared across jobs (where all other attributes match) etc.
The dependencies caches are shared between Jobs, if those Jobs have the exact same set of dependencies resolved.
I might want to clarify the docs a little:
If the setup-gradle
action were smarter, it could use a "layered" approach where the set of dependencies that is commmon between Jobs would be extracted into a shared entry, and a second entry would have the dependencies that are unique to the Job.
I did try to have a separate cache entry for each dependency, for maximal cache storage efficiency. This approach quickly led to 429 errors in the cache: GHA cache doesn't like too many small entries.
lol, yea, can see that being a challenge. Lots of complexities here. Thanks for continuing to evolve this.
Thank you both for investigating and explaining.
I understand that the caches are saved 'per job', and I wonder how that works with re-usable workflows?
Kotest uses a re-useable workflow to call Gradle. The Gradle workflow is called twice in the same job: once to generate a JVM API dump (using BCV), and then again to run all Kotlin Multiplatform tests.
BCV only needs JVM dependencies, while the KMP tests require all KMP dependencies.
How would the dependencies cache for the job be populated? Would a single one be re-used for both invocations of the re-usable workflow, or a separate one each? Could it cause the dependency cache to be unnecessarily discarded?
Here's a Kotest run that shows the problem.
Specifically, this workflow snippet:
jobs:
validate-api:
name: Validate API
if: github.repository == 'kotest/kotest'
uses: ./.github/workflows/run-gradle.yml
secrets: inherit
with:
runs-on: ubuntu-latest
ref: ${{ inputs.ref }}
task: apiCheck
validate-primary:
name: Validate on primary runner
if: github.repository == 'kotest/kotest'
needs: validate-api
uses: ./.github/workflows/run-gradle.yml
secrets: inherit
with:
runs-on: ubuntu-latest
ref: ${{ inputs.ref }}
task: check
...doesn't store cache information on the second job, many entries such as:
Entry: /home/runner/.gradle/caches/modules-*/files-*/*/*/*/*
Requested Key : gradle-dependencies-v1-a702ec8f84890c954ac77cc8b9b7e1be
Restored Key : gradle-dependencies-v1-a702ec8f84890c954ac77cc8b9b7e1be
Size: 445 MB (466172742 B)
(Entry restored: exact match found)
Saved Key :
Size:
(Entry not saved: referencing 'Gradle User Home' cache entry not saved)
...perhaps due to Gradle User Home cache not changing (same SHA, same OS, ...):
Entry: Gradle User Home
Requested Key : gradle-home-v1|Linux|run-tests[2c202e428a822dfa2e29a4f7ab04d507]-0054608c69772bd253aec319bedc2231983e13f1
Restored Key : gradle-home-v1|Linux|run-tests[2c202e428a822dfa2e29a4f7ab04d507]-0054608c69772bd253aec319bedc2231983e13f1
Size: 1 MB (791418 B)
(Entry restored: exact match found)
Saved Key :
Size:
(Entry not saved: cache key not changed)
Ahh....
This cache key for GUH: gradle-home-v1|Linux|run-tests[2c202e428a822dfa2e29a4f7ab04d507]-0054608c69772bd253aec319bedc2231983e13f1
contains run-tests
as the job name - would expect it to change, to be validate-api
or validate-primary
.
run-tests
is the job name in the reusable Github workflow that setups & executes Gradle via uses: ./.github/workflows/run-gradle.yml
.
So it would appear that setup-gradle
has the immediate job name, but not the overall context and we're colliding as a result.
(this is the issue I thought was a race condition in the initial part of this ticket)
@cloudshiftchris Thanks for investigating further. This indeed looks like a bug, but I'm not sure how to fix it.
The full identifier of the "Job" is run-tests[2c202e428a822dfa2e29a4f7ab04d507]
: the part in brackets should uniquely identify different jobs with the same name, generated here. It takes into account the following:
Unfortunately this doesn't differentiate between 2 Jobs with the same name in the same workflow, which is what you have.
Although validate-api
("Validate API") and validate-primary
("Validate on primary runner") are declared in the workflow file and are visible in the UI, there's no obvious way to access these values during workflow execution. If you can find a way, then it should be possible to include these in the cache key.
Oh wow, this (getting the full job name) is way more complicated than it should be.
There are a couple of open issues on this:
Setup repo to reproduce this (one workflow, two jobs that call a reusable workflow) with lots of debugging - the only place the full job name is evident is in an early job entry: Complete job name: build-first / run-tests
- none of the contexts or env vars have the build-first
portion, only the current job name of run-tests
.
This snippet returns something that may be useful (it has build-first / run-tests
) - haven't yet grokked how to cleanly correlate the returned json to the current job, and don't know if API calls like this are appropriate for setup-gradle
- name: grab job id
run: |
set -x
job_url="https://api.github.com/repos/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID/jobs"
job_json=$(curl -v -s -H "Authorization: token ${{ github.token }}" $job_url | jq)
job_id=$(curl -v -s -H "Authorization: token ${{ github.token }}" $job_url | jq -r '.jobs[] | .id')
echo $job_json
echo $job_id
Log output from run here.
Would using the concurrency group be possible, and useful?
It's possible to encode all inputs of the workflow as a concurrency group using ${{ join(inputs.*) }}
.
Good thoughts.
Looked at using input
- it doesn't fully address the issue. Collisions would still occur if the same reusable workflow was invoked more than once with the same inputs.
Tried with concurrency-group
- not seeing that visible anywhere in the available information; less that ideal for users to have to configure that (and know to do so) to avoid cache collisions.
What we really need is the full job name, as shown in the UI - validate-api / run-tests
, validate-primary / run-tests
- to indicate that "cache is for this specific job".
Collisions would still occur if the same reusable workflow was invoked more than once with the same inputs.
There would be a collision, but would it be a problem? Am I right in thinking that the resulting cache entry will unnecessarily be updated, but at least it should contain the same dependencies?
Also, is it likely that a re-useable workflow with the same inputs will be called twice?
Am I right in thinking that the resulting cache entry will unnecessarily be updated, but at least it should contain the same dependencies?
What happens on a GUH collision is that the cache is not updated (for the second job) - it loads cache entries as they existed for the first run, but doesn't store anything on the premise that, because the job key is the same, everything else must be the same (which would work if the job was uniquely identified).
Could imagine scenarios where reusable workflows are called multiple times with the same inputs (or no inputs, if all one desires is the side-effects). There's drawbacks to using inputs
- if they change it would invalidate the Gradle cache, which isn't the intention as its the same job (but we don't have a full job id/name to determine that).
The cache entry for a key is immutable once written. So there's no way to "update" the cache entry without creating a new cache key. This is why we try hard to create a unique cache key for each Job execution (using workflow-name + job-name + matrix-values + Git SHA).
To avoid a massive explosion of cache usage, we first extract any content that we think might be shared between cache entries. These extracted entries are keyed only on their content, so you'll only get sharing if the content is exactly the same between to Job runs. This is commonly the case for subsequent runs of the same Gradle build (same Job), and less commonly occurs for different Jobs with similar content.
@cloudshiftchris I'd be interested to see how things work if the calling Jobs (validate-api
, validate-primary
somehow had different Matrix parameters defined (even if these aren't used). The cache key should include a hash of the Matrix values for a Job execution, but I'm not sure if these are also lost in a reusable workflow.
It's worth reading How Gradle User Home caching works, as it describes the properties of the GitHub Actions cache that led to the current model of caching in setup-gradle
.
Oh wow, this (getting the full job name) is way more complicated than it should be.
:-)
There are a couple of open issues on this:
- https://github.com/orgs/community/discussions/8945
- Reusable Workflows - predefined variables load for the called job actions/runner#2577
Setup repo to reproduce this (one workflow, two jobs that call a reusable workflow) with lots of debugging - the only place the full job name is evident is in an early job entry:
Complete job name: build-first / run-tests
- none of the contexts or env vars have thebuild-first
portion, only the current job name ofrun-tests
.
Thanks so much for taking the time to research. This really helps me out.
This snippet returns something that may be useful (it has
build-first / run-tests
) - haven't yet grokked how to cleanly correlate the returned json to the current job, and don't know if API calls like this are appropriate forsetup-gradle
- name: grab job id run: | set -x job_url="https://api.github.com/repos/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID/jobs" job_json=$(curl -v -s -H "Authorization: token ${{ github.token }}" $job_url | jq) job_id=$(curl -v -s -H "Authorization: token ${{ github.token }}" $job_url | jq -r '.jobs[] | .id') echo $job_json echo $job_id
Log output from run here.
I think we might be able to adapt something like this to run inside the action. While I'd prefer not to have to make an API call, if it can provide all of the context required to create a unique cache key, it could be worth it.
Can you try adding some matrix parameters to your test and see if they are also included in the JSON? I'm currently accessing them like this, which works but ideally I wouldn't need this layer of indirection.
Ah, now I see that I can only obtain the set of all Jobs running, and there's no way that I can tell to determine which one matches the currently executing Job!
https://stackoverflow.com/questions/71240338/obtain-job-id-from-a-workflow-run-using-contexts
Yea, that's where I ran into the brick wall as well, its unclear / imprecise how to match things up :(
@bigdaz
Can you try adding some matrix parameters to your test and see if they are also included in the JSON? I'm currently accessing them like this, which works but ideally I wouldn't need this layer of indirection.
Tried two variations on matrix jobs here; findings:
1) Reusable workflows (see sample below) show no usable indication of the matrix values; as with the job id above the matrix information is only available in the "Complete job name" debug output and the "jobs" api calls that we're unable to correlate with. This include using the same method as setup-gradle
uses - its likely (subtly) broken for reusable workflows.
build-matrix:
strategy:
matrix:
a-matrix-dimension: [ 10, 12 ]
uses: ./.github/workflows/run-gradle.yml
- name: Dump matrix context
env:
MATRIX_CONTEXT: ${{ toJson(matrix) }}
run: echo "$MATRIX_CONTEXT"
##[group]Run echo "$MATRIX_CONTEXT"
[36;1mecho "$MATRIX_CONTEXT"[0m
shell: /usr/bin/bash -e {0}
env:
MATRIX_CONTEXT: null
##[endgroup]
2) Jobs that have steps (sample below) can directly see the matrix context:
build-matrix-2:
strategy:
matrix:
a-matrix-dimension: [ 10, 12 ]
runs-on: ubuntu-latest
steps:
- name: Dump matrix context
env:
MATRIX_CONTEXT: ${{ toJson(matrix) }}
run: echo "$MATRIX_CONTEXT"
- name: Dump matrix context
env:
MATRIX_CONTEXT: ${{ toJson(matrix) }}
run: echo "$MATRIX_CONTEXT"
2024-08-01T14:07:21.1119211Z ##[group]Run echo "$MATRIX_CONTEXT"
2024-08-01T14:07:21.1120721Z [36;1mecho "$MATRIX_CONTEXT"[0m
2024-08-01T14:07:21.1469240Z shell: /usr/bin/bash -e {0}
2024-08-01T14:07:21.1469770Z env:
2024-08-01T14:07:21.1470353Z MATRIX_CONTEXT: {
"a-matrix-dimension": 10
}
2024-08-01T14:07:21.1470948Z ##[endgroup]
2024-08-01T14:07:21.1873715Z {
2024-08-01T14:07:21.1875356Z "a-matrix-dimension": 10
2024-08-01T14:07:21.1875945Z }
After a bit more investigation, I'm convinced that we should be including the inputs for a reusable workflow in the Gradle User Home cache key, similar to how we currently include the matrix values. This would allow us to differentiate between different "instances" of the same Job in the workflow.
As far as I can tell, the only way to differentiate between different calls to a reusable workflow are via inputs: environment variables aren't automatically passed in, so I think it's fair to differentiate based on inputs.
I forked the reproducer to include more dimensions in the workflow:
Here's a run with the default configuration. Each of the * / run-tests (--scan)
jobs shares a cache key, as do each of the * / run-tests (--stacktrace)
jobs. This makes sense, since only the [--scan, --stacktrace]
matrix is being supplied in the workflow-job-context input parameter.
Here's a workflow run where the workflow inputs are explicitly added to the workflow-job-context parameter. Each of the Job executions has a unique Gradle User Home cache key, as desired.
Unfortunately, I can't (yet) find a way to access the inputs directly from the action implementation, and this needs to be configured by the user. I can access the matrix
values like this, but adding the inputs
here seems to have no impact. Grrrr.
So we have a solution for reusable workflows, but it requires a user to know this is required and to supply the inputs specifically to the action. (Note that passing in the "complete job name" value would also work: just need something that uniquely identifies the job execution).
Repro: https://github.com/cloudshiftinc/gradle-setup-action-cache-repro/actions/runs/10067337462
That repository consists of a small "Hello World" Gradle / Kotlin project, with two build files: 1) Build with Kotlin 1.8.22; 2) Build with Kotlin 1.9.25;
There's a GHA workflow that runs two steps, sequentially, using each of the above build files. Each step checks out the code, calls
setup-gradle
(v4 beta 1), and executes the respective Gradle build file (build.gradle.kts or build-second.gradle.kts).The second build "cleans up" artifacts it didn't use from the first build:
This isn't desirable in a multi-step setup - we'd want to retain all the cached entries and only cleanup at the end.
For reasons I don't currently understand the cache isn't updated after the cleanup in the second build, even though the cleanup removed entries:
The issue starts here where
setup-action
stores the current timestamp, and then in the post-build cleanup removes everything older than that. This assumes a closed-loop where it's a self-contained build, which doesn't work for shared caches (dependencies, wrappers, etc) that other builds will use/update/cleanup.Not sure what the best solution may be - perhaps loosen the cleanup windows to "older than a day"; not thrilled about making it configurable, as one wouldn't know what to reasonably put (is 10m too long? what if the execution time varies? and so on).
If there's a way to get the "start of the workflow" timestamp that may be more deterministic, though that still leaves holes where there are different workflows running that share those caches - one may wish to save the shared caches for longer periods (e.g. workflows that use those caches that run weekly, for example).