CI can fail due to docker time outs

[sjspielman]

Recent builds are failing on Circle CI because of time outs:

Try out a larger resource class or running tests in parallel to speed up job execution. Upgrade your pricing plan to take advantage of longer max job runtimes.

context deadline exceeded

Looks like we are surpassing the three hour limit, and there is no performance degradation right now with Circle CI that could affect the run time. As more analyses get updated and pub-ready figures created, my guess is we will keep hitting this time limit moving forward.

From Circle CI docs:

Note: Jobs have a maximum runtime of 1 (Free), 3 (Performance), or 5 (Scale) hours depending on pricing plan. If your jobs are timing out, consider a larger resource class and/or parallelism. Additionally, you can upgrade your pricing plan or run some of your jobs concurrently using workflows.

[sjspielman]

Tagging @jaclyn-taroni @jashapiro

[jaclyn-taroni]

We're not upgrading to Scale, so that option's off the table.

In my opinion, the problem is that we're not seeing a benefit from the Docker layer caching right now and then 2 of 3 hours is eaten up by building the image. That might be because CI is running less frequently than earlier in the project when PRs were more frequent. It might be because how one configures that correctly has changed over the life of the project.

So some options (and these are not mutually exclusive):

• Read up on DLC. Listed first cause I think it's step 1!
• Time for some Docker "housekeeping" to reduce the build time. We've generally put new packages at the end for convenience for a bit now. This is where I'd start with respect to changes to the code base.
• I do think there are things at the end of the steps that have outlived their usefulness
• Commenting out steps that are not currently headed to the manuscript (don't love that)
• Parallelism, which I expect to be complicated by dependencies between modules.

[jaclyn-taroni]

I'm going to try altering #1224 because why not

[jaclyn-taroni]

Noting here that for the jobs that are rerunning after timing out Docker layer caching appears to be working as expected.