search

microsoft / knossos-ksc

Compiler with automatic differentiation
Other
45 stars 10 forks source link

Reduce cache misses on MLIR build #719

Open awf opened 3 years ago

toelli-msft commented 3 years ago

Whenever we change version of LLVM we need to build master to ensure that new branches will be able to use master's cache of LLVM. If we don't build master then PRs of new branches will spend 80 minutes building it themselves!

@cgravill says

There are involved compulsory rules on cache isolation: https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#cache-isolation-and-security

We can still roll our own version with storage and our own rules, I generally prefer to use what's been provided but it's behaviour isn't ideal for us.

Instead of rolling our own we can get almost all of the benefit by just remembering to rebuild master once an LLVM version change has been merged. How can we prompt ourselves to do so?

It would be nice to put a comment in https://github.com/microsoft/knossos-ksc/blob/2b17ca55f618457b6c423d167b9bc605a9d11142/etc/llvm-branch.txt, but the consumers would have to be updated.

cgravill commented 3 years ago

One option is to just set a scheduled build on master. Then the misses won't last long.

cgravill commented 3 years ago

Just for information, the equivalent feature on GitHub Actions has the same security feature: https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache

toelli-msft commented 3 years ago

We have a few places that $(cat $KNOSSOS/etc/llvm-branch.txt). If instead we rename it to llvm-branch.sh and change its contents to

#!/bin/sh

# After merging a change to the LLVM version to master please
# rebuild master else every new branch will build all of LLVM.
#
# See https://github.com/microsoft/knossos-ksc/issues/719

echo a88371490dae07450169bfc759efa6e443624d4a

and replace $(cat $KNOSSOS/etc/llvm-branch.txt) with $(sh $KNOSSOS/etc/llvm-branch.sh) then that will probably suffice.

cgravill commented 3 years ago

For the build we might be able to trigger it with something like this: https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#paths

# UNTESTED!
trigger:
  branches:
    include:
    - master
  paths:
    include:
    - etc/llvm-branch.txt