Action Buildtime optimizations

[lmangani]

The build_wheels.yml action is huge and could benefit being split into multiple stages to gain significant cross-build speedup. Currently all builders seem to all run in parallel, leveraging neither git submodule caching or library artifact sharing.

I might be wrong as python is not my forte, but assuming the same .so could be used by all binding builders on the same arch (is this the case?) having the clickhouse/chdb library build stage and the python binding stage split/chained could save hours on releases for the same OS/arch over a matrix of python versions.

Note: not a request - posting to find a way to contribute and help out

[auxten]

Since the compilation of the Clickhouse part is very time-consuming, it cannot be completed within the 6h limit of the github action without a ccache hit. So if you modify a flag like avx2 that will affect ccache, the first github action compilation will indeed fail, but this time you will get a ccache that speeds up the second compilation. But in most cases, the flag in the compile phase will not be modified, so due to the existence of ccache, the general github action can be completed within 20 minutes. An ideal github action can split the part of compiling cpp code into two common steps of macos and linux, but if the flag of compile is modified, this step will still inevitably time out.

[lmangani]

Makes sense. Then its about small step improvements. For example, a required job could perhaps provide same contribs cache to all platform builders and save ~15m each run already: untested example

[auxten]

Clickhouse builds contribs with its own build system which means some compile flag changes will also influent contribs. I don't think the contribs cache will make things better.