Is it possible to cache digested Phoenix output?

[tomekowal]

We have a Phoenix app that is mostly backend but still has two or three web pages. The JS very rarely changes, but significant part of build time is consumed by recompiling JS files.

Would it be possible to skip all that work if package-lock.json and md5 sum of source files did not change between builds?

[gjaldon]

I think that's possible. What we do now is we store node_modules in the cache_dir so that we just copy the packages from there before running npm install since what's stored in the cache_dir get carried over in the next builds.

You could do the same in the compile file by:

• check if package-lock.json exists in cache_dir
• if it does, compare package-lock.json in $cache_dir with package-lock.json in app root.
  • if equal, skip the build command of your build tool
  • if not equal, then run the build command
• if it doesn't, run the build command
• copy package-lock.json into the $cache_dir

You could probably do the same with the compiled js file. Store it in the cache dir so you can copy it in later builds so that subsequent builds would be faster. That might help speed up subsequent builds where there were changes in package-lock.json.

Let me know how that pans out.

[tomekowal]

I did some measurements locally, and I came up with this:

npm install  10,97s user 2,22s system 111% cpu 11,883 total
npm run deploy  7,34s user 0,60s system 150% cpu 5,267 total

npm install takes its time even when running on the already compiled cached directory. It looks like caching compilation phase gives fewer benefits than not running npm install. Unfortunately, that is outside of configurable compile script.

I wonder if the strategy you've mentioned of comparing package-lock.json couldn't be used here: https://github.com/gjaldon/heroku-buildpack-phoenix-static/blob/3e414fcfe8c2bfb6a01f243bbfa2ef8984f37fa5/lib/build.sh#L117 to eliminate calls to npm install.

I could make a PR if you like the idea.

[gjaldon]

A PR would of what you described would be great, @tomekowal!

Also, turns out there's a npm ci command now: https://docs.npmjs.com/cli/ci.html https://medium.com/@lusbuab/npm-ci-a-command-you-should-know-d8d67847884c

I guess we could use that npm ci when there is a package-lock.json present and use npm install when it's not there.

[tomekowal]

Or use only npm ci and error out when there is no package-lock.json. It looks like the sole purpose of npm ci is to force consistency and ensuring correct lock file. Doing:

if package.lock do
  npm ci
else
  npm install

defeats its purpose :)

Also, it discards cached node_modules to build from a clean state every time. In my project, it takes twice more time. added 1085 packages in 24.014s

It is a tradeoff between speed of npm install on cached node_modules and consistency of npm ci always starting with clear state and forcing correct lock file.