search

MatrixAI / js-db

Key-Value DB for TypeScript and JavaScript Applications
https://polykey.com
Apache License 2.0
5 stars 0 forks source link

Integrate ccache for caching native compilation #44

Closed CMCDragonkai closed 3 months ago

CMCDragonkai commented 1 year ago

Specification

Native compilation takes quite some time right now (17 minute jobs for nix, 27 minute jobs for windows), this is because node-gyp uses makefiles. Makefiles rely on file timestamps to know when to recompile.

The problem with file timestamps is that on the CI/CD, every time it clones the source code, or fetches the source code, the timestamps are in fact changed.

This causes node-gyp to recompile everything even if the ./build directory is cached. Therefore caching the ./build directory is not useful.

According to https://stackoverflow.com/questions/48522007/is-it-possible-to-rebuild-only-updated-files-in-gitlab-ci Gitlab's CI/CD caching system does in fact preserve modification times of the cached files. However again this is not sufficient.

There's a project called ccache https://ccache.dev/manual/4.6.1.html, that sits in front of the C/C++ compiler. When combined with gitlab's cache, it is possible to cache the compilation. Here's an example of using the ccache https://stackoverflow.com/a/71798144/582917. It appears the idea is that ccache caches the intermediate build objects and stores this in a directory. This directory is then cached by gitlab CI/CD. Upon re-running, node-gyp would rerun make, and since the timestamps changed, it will continue to try and execute the C/C++ compilers. But ccache intercepts this and finishes the execution.

It is possible to integrate ccache with node-gyp by sticking ccache in front of any calls to the $CC. However this may need to be experimented on with nix, windows and macos. See: https://stackoverflow.com/questions/25146976/can-i-get-npm-gyp-to-use-ccache

When attempting to use ccache in nix, it may work fine when running inside nix-shell, but then if this was an application that we were building and we were using nix-build, then the build isolation will ignore any usage of ccache. It would be necessary to "mount" the ccache directory into the build environment somehow. See https://github.com/NixOS/nixpkgs/blob/104f8a0e1a17a894b320f86add409d9aebb36fe4/pkgs/top-level/all-packages.nix#L9007-L9026 and https://discourse.nixos.org/t/caching-local-compilation-results-for-all-compiled-packages/3516

While developing a derivation, I’ve had luck just setting the CCACHE variables directly and turning sandboxing off. This allowed me to temporarily speed up iterations of builds, then turn it off before submission to Nixpkgs.

Alternatively we always maintain that native addons should be put into JS libraries, and thus never need to use nix-build.

Additional context

Tasks

  1. ...
  2. ...
  3. ...
CMCDragonkai commented 1 year ago

This is all because we're still using JS toolchain to build stuff, but they haven't migrated to a Nix based toolchain. Perhaps one day the deno system will make nix first class.

CMCDragonkai commented 1 year ago

This thread will be relevant https://github.com/MatrixAI/js-logger/pull/21#issuecomment-1191317203 regarding caching dealing with new branches and new tags and security between protected and unprotected branches.

CMCDragonkai commented 3 months ago

We are probably going to migrate to Rust fully so we won't really bother with integrating ccache. Plus moving towards GitHub Actions over GitLab CI. So the above doesn't really matter.