Consider whether prebuilds would be useful to speed up build times

[StevenMaude]

GitHub allows prebuilding of dev containers to speed up the start time.

If it currently takes more than 2 minutes to create a codespace for a repository, you are likely to benefit from using prebuilds.

• Would a prebuild be necessary? Will the dev container we use be annoyingly slow to start without?
• How much faster would a prebuild make starting a new codespace?
• What would the prebuild storage cost be? Is that acceptable for the speed up we gain?
• How do prebuilds interact with template repositories, and user-/organization-owned codespaces?
  • Can we configure prebuilds for newly created repositories from the research-template repository?
  • Who incurs the storage cost (user or organization)?

[StevenMaude]

Reading around, I think that a newly created repository from a template doesn't share the same prebuild; each repository has its own independent setup.

So if users want to enable prebuilds, they should do it themselves. However, prebuilds incur a storage cost that would continue to accrue, even when the project might be complete. How would we prune unneeded prebuilds?

[StevenMaude]

Rough timing

I forked the repository as of commit opensafely/research-template@92d82890dc8e4b72de034724a7254170c79cbf79 and roughly timed the startup times with a stopwatch, starting when I pressed the "Create codespace on main" button and recording the approximate time when:

• the terminal prompt appeared
• sudo rstudio-server start ran and the next terminal prompt appeared

With prebuild

Terminal start (seconds)	R Studio start (seconds)
47	80
51	88
58	88

Without prebuild

Terminal start (seconds)	R Studio start (seconds)
220	263
201	241
213	251

Summary

Without a prebuild, the start times to get usable codespaces are (roughly) 160 seconds slower. This isn't blazingly fast, but nor does it feel unacceptably slow; it's a similar time to making a coffee (or other preferred drink).

Even with a prebuild, there is some startup delay.

[StevenMaude]

Rough costing

Storage costs

The storage costs depend on the:

• cost per GB-month
• prebuild size
• number of geographic regions that you store the prebuild in (default is all, which is currently 5)
• number of versions you retain (default is 2)

I think the only place you can see this for prebuilds is when it is logged in the billing usage. There doesn't seem to be any other way.

As an estimate for now, let's assume the storage per single build is the same as the codespace, which is 6.26 GB.

The storage cost is currently $0.07 per GB-month, assuming that the cost is the same as for codespace storage.

A single build would cost $0.43 per month to store. With the default settings would cost 10 * $0.43 per month, so an upper cost of $4.30 per month. This cost could be reduced by reducing to fewer regions.

Edit: billing takes a day or so to show up. Looking at my billing after having stored a prebuild for about an hour, I got charged 0.07 GB-month usage. Scaling that usage from an hour to a month, that's 0.07 24 30, corresponds to about 50 GB stored for a month for 10 builds, so about 5 GB per build, and not 6.26 GB. So the actual storage cost would be about 80% of the cost, but does not change the conclusions.

Build costs

Prebuilds use GitHub Actions; private repositories using prebuilds will incur a GitHub Actions usage cost. Public repositories do not incur a cost.

For opensafely/research-template@92d82890dc8e4b72de034724a7254170c79cbf79, prebuild for all regions took about 18 minutes.

Per build for private repositories, this is about $0.144 with a 2-core runner ($0.008 per minute).

The cost then depends on how frequently users push to the main branch, which will depend on:

• how often they are working on the repository
• if they are using branches and pull requests for development, or pushing to the main branch directly

This can be fixed by building on a regular schedule; say, daily or weekly.

The build cost could be of the same order as the storage cost, if builds are frequent.

Rough estimate

So let's say, for a single repository:

• storage cost of $1-$5 per month
• very rough estimated additional build cost of $4.32 per month for private repositories (assuming one build per day, either scheduled or triggered by a push to the main branch)

For public repositories, about $1-$5 per month. For private repositories, about $5-10 per month.

For comparison, the cost of using a 2-core codespace for 20 hours per week, is currently about $16.

So is the decrease of time worth the additional cost, I'm inclined to say: probably not, unless it's a public repository where the costs are smaller. However, this might warrant a re-review if the codespace build time ever increases a lot.

[StevenMaude]

Overall, it seems that prebuilds aren't a big enough benefit to use right now.

But to summarise answers to the questions in the opening comment here:

• Would a prebuild be necessary? Will the dev container we use be annoyingly slow to start without?

No, it is not necessary, and the start time is reasonable.

• How much faster would a prebuild make starting a new codespace?

About 160 seconds faster.

• What would the prebuild storage cost be? Is that acceptable for the speed up we gain?

The costs are likely in the range of $1-10 per month per repository.

• How do prebuilds interact with template repositories, and user-/organization-owned codespaces?

• Can we configure prebuilds for newly created repositories from the research-template repository?

It does not seem possible. The configurations appear to be independent. So a prebuild needs configuring per repository.

• Who incurs the storage cost (user or organization)?

Assume this is the organization.