search

maxitg / SetReplace

C++/Wolfram Language package for exploring set and graph rewriting systems
MIT License
216 stars 43 forks source link

Only download Mathematica/Wolfram Engine once per workflow #579

Open maxitg opened 3 years ago

maxitg commented 3 years ago

The problem

Due to tests parallelization #578, we are now downloading the docker image with Mathematica 4 times, which is redundant. We should figure out how to only download it once thus saving 3 minutes' worth of credits. If we download it prior to starting the testing job, we will also reduce the workflow duration by 1 minute.

Possible solution

  1. Take WL out of the Docker image and put it into the Workspace for all nodes to use (tricky because it needs to stay activated).
  2. We can then add another wolfram-engine-download job which is going to put it into the workspace.
  3. The build/test jobs will then just need to put symbolic links/copy relevant files which should be much faster.

Alternative solutions

Circle CI has docker layer caching which might be useful here, however, it is not available on the free plan, and even if it was, it's quite expensive costing more credits than the entire WL testing job.

aokellermann commented 3 years ago

Documentation for attach_workspace states that:

The full contents of the workspace are downloaded and copied into the directory the workspace is being attached at.

Therefore, if we persist the wolfram installation and attach on our four parallel jobs, it will download the wolfram installation on each one. DLC looks like it could be promising though.

maxitg commented 3 years ago

We should even be able to cache it between builds similar to how you cache Python modules.