WIP: Travis CI docker env

[matthewdehaven]

Attempt to decrease total runtime of Travis runtimes by using a docker image with validate, sbcl, and ccl already installed, rather than building and installing those programs for each job. Decreasing total runtime of the tests may be valuable if a time allowance limit becomes an issue with the Travis CI tests.

[matthewdehaven]

@rpgoldman I was not able to decrease the total runtime of the tests by much. A large chunk of startup time in the tests was taken up by each job building validate and installing sbcl or ccl, about 50s in total for each job. I modified the configuration to run the tests in a docker container which already had validate built and sbcl and ccl installed. However, it now takes 20s per job to pull the docker image. So the total time savings is only ~30s per job, ~15 minutes total over the 32 jobs. The total runtime is usually around 2h45m. So maybe a ~10% decrease in total runtime.

It's not clear to me whether or not that time savings is worth including docker in the process.

Comparing two actual runs showed only a 7 minute decrease in runtime, I guess just due to some amount of normal variance in the test runtimes on the Travis machines.

Without docker image: https://travis-ci.com/github/shop-planner/shop3/builds/225378869

With docker image: https://travis-ci.com/github/shop-planner/shop3/builds/226010118

Startup time per job without docker image:

Step	Time (s)
VM Startup?	16
Git Clone	1.3
Git Submodules	7.6
rvm use	0.66
Lisp Install	21
validate compile	31
Lisp Compile	4.7
Total	82

Startup time per job with docker image:

Step	Time (s)
VM Startup?	16
Docker Start	3
Git Clone	1.3
Git Submodules	7.6
rvm use	0.66
Docker Pull	17
Lisp Compile	7.9
Total	53

[rpgoldman]

I wonder if the best thing would be to do the lisp compile once, save the image, and then load it afterwards? There's a way to do this with ASDF, but I haven't used it in years.

But given that I start up shop over and over, saving the image and just loading it, instead of loading the fasls over and over might be faster.

The validate compile I'm less sure about -- I don't know how that would be shared, but it seems like it ought to be possible.

[matthewdehaven]

I think that makes sense to create an image once and re-use it, but I am unsure of how to do that in Travis. My initial thought was that a build stage could be added to the Travis CI configuration. That build stage could build the image, then pass that on to the testing stage.

But Travis doesn't seem to have a way to pass build artifacts to later stages. I couldn't find any way to do pass files between stages in the docs, besides upload it somewhere, then redownload it in the later stage. With a lisp image that may be 50 MB, I'm not sure that will save much time. Plus, it seems you have to figure out your own place to upload and download the file.

I feel like I must be missing something, because building some objects once and using them throughout all the tests seems like a common task. But the first SO result I found when looking for solutions suggests abusing github releases to share files between build stages. So maybe it really is something that is not straight forward with Travis.

https://stackoverflow.com/questions/53931069/pass-information-between-stages-in-travis-ci

I'm not really sure how to accomplish this goal of building the image once and re-using it.

[rpgoldman]

I guess we could make a docker image with the built lisp images and then load that? Seems yucky.

Also, in the docker script is there any way to save a layer so that validate doesn't have to be built repeatedly? It's pretty stable (indeed, it's there because upstream moved in incompatible directions).

[matthewdehaven]

Yeah, docker wouldn't need to rebuild validate every time. I agree that using a docker image with the built lisp images does seem not very nice. I'm also not sure that it will end up reducing the run time much since there seems to be quite a bit of the Travis environment startup time that fixed and unrelated to the lisp compile.

Another solution idea I was thinking of to reduce the time spent on startup is decrease the number of jobs. There are 32 jobs right now. 4 of those jobs take about ~30 minutes each. The others all seem pretty quick. If the other 28 jobs could be grouped together in 1 or 2 jobs that take less than 30 minutes each, the time to finish would be the same, but the overall runtime would decrease by reducing the number of times the startup is repeated.

[rpgoldman]

Grouping the jobs sounds reasonable -- I split them out because some of them took so much time, and I was having timeout issues.

[rpgoldman]

I think this is OBE because Travis has been supplanted by GitHub actions.