adoptium / containers

Repo containing the dockerfiles and scripts to produce the official eclipse-temurin containers.
https://hub.docker.com/_/eclipse-temurin/
Apache License 2.0
204 stars 87 forks source link

Add request caching to Dockerfile generator #536

Closed rassie closed 2 months ago

rassie commented 2 months ago

When developing parts of the Dockerfile generation pipeline, and thus regenerating Dockerfiles over and over, many unnecessary HTTP requests are being made to https://api.adoptium.net, since the data is mostly unchanged. Adding a request cache with an expiration time of an hour allows this regeneration to run in under a second in most case, while not compromising on the correctness.

gdams commented 2 months ago

This will exacerbate the timing window already present in this logic, as the retrieval of the assets at the centre of the configuration/version/image_type loop is requested as the 'ga' which may change during the script run if you were unfortunate to be running while a Temurin release is underway - though I accept that is unlikely. The failure would be a skewed set of release images.

We could implement a parameter to skip cache? Then maybe say that for workflow_dispatch runs (e.g manual ones) it skips it? WDYT?

rassie commented 2 months ago

Hmm. I thought that workspace is cleaned before a build and thus the cache file is gone (i's in .gitignore), so that I assumed this patch would only be valid for local development, but not for CI. Do you keep files between builds?

gdams commented 2 months ago

Hmm. I thought that workspace is cleaned before a build and thus the cache file is gone (i's in .gitignore), so that I assumed this patch would only be valid for local development, but not for CI. Do you keep files between builds?

hmm yes this is a good point, this only affects local development. In which case I'm happy

tellison commented 2 months ago

Yes, I'm happy to leave as is based on the original post that ... "When developing parts of the Dockerfile generation pipeline, and thus regenerating Dockerfiles over and over..."

This would not affect the usual usage. I did momentarily consider whether it should have a guard only set for local development, but that seem unnecessary. Deploying for production and in GHA it is a single run through and we wouldn't be making repeated calls to the same API anyway.