search

ratt-ru / QuartiCal

CubiCal, but with greater power.
MIT License
7 stars 4 forks source link

Cache NUMBA kernels between CI runs #279

Closed sjperkins closed 5 months ago

sjperkins commented 1 year ago

Closes #278

JSKenyon commented 1 year ago

This looks awesome! I will probably move this to the dev branch before merging.

sjperkins commented 1 year ago

This looks awesome! I will probably move this to the dev branch before merging.

Cool. Need to prod it a bit to see if it works.

sjperkins commented 1 year ago

Hmmm, I was fairly sure I mkdir'd that

image

sjperkins commented 1 year ago

So the kernel caching does not seem to be improving the test suite run time. even though kernel caches are created: https://github.com/ratt-ru/QuartiCal/actions/caches. This would also seem to suggest NUMBA_CACHE_DIR is respected.

JSKenyon commented 1 year ago

So the kernel caching does not seem to be improving the test suite run time. even though kernel caches are created: https://github.com/ratt-ru/QuartiCal/actions/caches. This would also seem to suggest NUMBA_CACHE_DIR is respected.

Is respected, or isn't? Might need to rerun the tests a few times - I have muddied the waters by merging in main. I do think that there is probably something which can be done - will take a closer look at the end of the week.

sjperkins commented 1 year ago

So the kernel caching does not seem to be improving the test suite run time. even though kernel caches are created: https://github.com/ratt-ru/QuartiCal/actions/caches. This would also seem to suggest NUMBA_CACHE_DIR is respected.

Is respected, or isn't?

I think it is respected -- The caches are about 11MB.

Another thought occurred, the cached kernel modification times are probably earlier than the checked out python code -- this might trigger recompilation: https://numba.readthedocs.io/en/stable/developer/caching.html

The cache is invalidated when the corresponding source file is modified.

Edit: Referenced the main article on caching, rather than the cuda article.

sjperkins commented 1 year ago

Another thought occurred, the cached kernel modification times are probably earlier than the checked out python code -- this might trigger recompilation: https://numba.readthedocs.io/en/stable/developer/caching.html

The cache is invalidated when the corresponding source file is modified.

Unfortunately it looks like it is the case that the timestamp is only the input to the cache key (at least as of Aug 22): https://numba.discourse.group/t/cache-behaviour/1520

I would like to propose to move away from invalidating the cache index based on the timestamp of the file, and use only the code+closure signature of the function itself. Would anyone see a problem with that? When I say code+closure signature I mean the exact same information that is used to select the overload from within the cache.

So this approach doesn't seem viable.

JSKenyon commented 1 year ago

So this approach doesn't seem viable.

Ah unfortunate. Perhaps there will be progress upstream at some point.