python / core-workflow

Issue tracker for CPython's workflow
https://mail.python.org/mailman/listinfo/core-workflow
Apache License 2.0
93 stars 60 forks source link

Enable ccache / sccache on our Github CI Actions #436

Open gpshead opened 2 years ago

gpshead commented 2 years ago

This should significantly reduce build latency, particularly rebuild latency when minor changes are pushed in PR branches.

It can help across builds, but as the main branch point of a given PR branch is not synchronized how much it helps there will be limited. A lot more likely to help on release branches for bugfix backports as C code in those trees doesn't evolve quickly.

https://github.com/hendrikmuhs/ccache-action has instructions. sccache even claims to support Windows (I've never used it).

ammaraskar commented 2 years ago

Looks like we even have the ccache action already hovering around in the workflow from here: https://github.com/python/cpython/pull/29935/files

Just need to use it for the C building bits.

ammaraskar commented 2 years ago

Actually, since the workflow uses the approach that adds the ccache folder to PATH:

https://github.com/python/cpython/blob/788154919c2d843a0a995994bf2aed2d074761ec/.github/workflows/build.yml#L67-L68

      - name: Add ccache to PATH
        run: echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV

we might already be using ccache for all the subsequent steps including Build CPython.

ammaraskar commented 2 years ago

Yeah, we definitely already are thanks to @tiran. Here's a comparison of the same commit built on python/cpython with a full cache vs my personal fork with no cache:

python/cpython

original build time

ammaraskar/cpython

fork build time


I think the action items here are adding the ccache action to the docs and mac build to speed them up and maybe looking at swapping out for sccache so it can be used on Windows as well.

gpshead commented 2 years ago

It also looked like sccache can be used with Windows if someone win-familiar could figure out how that needs to integrate with its build system.

erlend-aasland commented 8 months ago

We've been using ccache in our CI for a while. Should we keep this open for the Windows CI?