Open yujincheng08 opened 1 year ago
FYI, the first build: https://github.com/LSPosed/Metagisk/actions/runs/5209660209/attempts/1
And the second build (re-trigger): https://github.com/LSPosed/Metagisk/actions/runs/5209660209
And there are 1440 C/C++ misses!
Here you are the log from the first build, and you can see there are tons of rate limits after a specific step.
That drastically drops the hit rate of the second build.
There are very little thing we can do for this. Sccache & Ghac are designed like this: too much small object will lead to rate limit.
Try setup a s3 bucket if the speed is critical to you.
Ok. Then I will use sccache locally and then upload to GitHub action cache like I previously did.
https://github.com/topjohnwu/Magisk/pull/7052 seems a nice idea! We can add this feature in sccache-action
too:
@Xuanwo is that something you could implement ? :)
In order to work around this, I had to implement my own sccache logic and not rely on this action, here it is if it helps anyone:
- name: Initialize sccache environnement
shell: bash
run: |
echo SCCACHE_CACHE=$(sccache --show-stats | grep Local | cut -d '"' -f2) >> $GITHUB_ENV
echo DATE_STRING=$(date +'%Y%m%d') >> $GITHUB_ENV
sccache --stop-server
- name: Recover sccache cache
uses: actions/cache@v3
with:
path: ${{env.SCCACHE_CACHE}}
key: sccache-cache-${{runner.os}}-0-${{env.DATE_STRING}}
restore-keys: sccache-cache-${{runner.os}}-0
- name: Start sccache
shell: bash
working-directory: ${{github.workspace}}
run: sccache --start-server
[...]
- name: Cleanup sccache
working-directory: ${{github.workspace}}
shell: bash
run: sccache --stop-server
runner.os
AND the date. If the cache for today has not been created, it will grab the cache from yesterday thanks to the restore-key
logic. A single cache file is used, not hundreds.On cleanup, cache action upload today's cache if it was not available
A 2h C++ build now takes less than 1 hour.
Hth,
See it live here: https://github.com/f3d-app/f3d-superbuild/blob/main/.github/actions/f3d-superbuild/action.yml
what about integrating this into sccache action itself ? :) maybe as an option ?
what about integrating this into sccache action itself ? :)
That would be great, but I've never written a line of typescript in my life. Maybe the maintainer will want to integrate such a change.
@Xuanwo would you be up for this task ? :)
(added a link to the action on our repo)
In order to work around this, I had to implement my own sccache logic and not rely on this action, here it is if it helps anyone:
Hi all, seeing the same issue with the file fragmentation. I've also run into the mtime
issue with large rust builds forcing long rebuilds.
Note we've also tried using a remote S3, though since github secrets can't be used on public Pull Request events, that option isn't viable for actually open source projects.
This seems like a straightforward and useful implementation.
First step run the server, recover cache location and current data, then stop the server.
Question to @mwestphal: Why is there a need to start the sccache server (given that it will start on the first compile request)?
Question for the sccache
core maintainers: Is there any reason that this behavior (saving local sccache files, and restoring) is not the default for github actions?
Why is there a need to start the sccache server (given that it will start on the first compile request)?
I need to recover the location sccache will look for the its own cache (SCCACHE_CACHE
), I need to start the server for that. I then stop it, and recover the cache from gha, then start it again.
@sylvestre here's a toy I made on a fork of this action. https://github.com/0o-de-lally/sccache-action/tree/local It sets an input "local" which defaults to true in my version.
It will just cache the contents into a bundle called "sccache". It's minimalist, there are no key options. It will instead delete the cache every time and overwrite it (github doesn't allow just updating a cache).
This suits my needs.
Anyone else that wants to try it it's a zero config:
- name: enable sccache
uses: 0o-de-lally/sccache-action@local
cc. @mwestphal
I found that my action's cache hit rate is too low (especially for c/c++). I enable debug log and saw tons of rate limits. How to solve this?