0xPolygonMiden / miden-base

Core components of the Polygon Miden rollup
MIT License
68 stars 42 forks source link

Improve CI times #903

Open Mirko-von-Leipzig opened 2 hours ago

Mirko-von-Leipzig commented 2 hours ago

CI test job times are becoming exceedingly poor, exceeding 10min and often 30min. Other jobs are all < 3 minutes.

Here are some recent workflow runs:

which are both running against the next branch with caching enabled and cache is hit. The bulk of the time is taken to compile internal crates again:

   Compiling miden-objects v0.6.0 (/home/runner/work/miden-base/miden-base/objects)
   Compiling miden-lib v0.6.0 (/home/runner/work/miden-base/miden-base/miden-lib)
   Compiling miden-tx v0.6.0 (/home/runner/work/miden-base/miden-base/miden-tx)
   Compiling miden-bench-tx v0.1.0 (/home/runner/work/miden-base/miden-base/bench-tx)
    Finished `test-release` profile [optimized + debuginfo] target(s) in 10m 05s

which occurs 3 times within the job. So the vast majority of the time is compiling despite the cache working.

What's interesting is that without the cache these multiple compilations don't occur. This CI run for #901 onto main has a full compile time of ~10minutes which it only pays once. The other two steps for some reason don't require a recompile. Compilation should be a once off; step 1 is build, and 2 & 3 are different test invocations. So it appears that this concept of build once, run twice is no longer working on next.

This probably indicates a change in workflow, features or dependencies between main and next which causes this triple compile.

Summary of initial findings:

Some initial thoughts:

Mirko-von-Leipzig commented 2 hours ago

The actual proof tests take 3 minutes to run. The non-proof tests are < 10s.

So despite the proof tests being slow, tests are only like 5% of the current time; the rest is all compilation.

On main this is more like 30% test, 70% build which is still not great but is better at least.