Currently, all containerized workloads have their image pull policy set to IfNotPresent.
This change was originally made to facilitate the e2e testing setup (where the image cannot be pulled due to a lack of a registry).
However, this default comes with a huge downside: repeatedly re-submitting a workload won't work (meaning, the old image will be re-used), unless the node image cache is cleaned manually (which first requires removing any stopped containers using the old workload image).
A more sensible solution would be to expose this image pull policy as part of the job submission options so that users can choose for themselves.
IMO, Always would be the safest default to prevent any surprises due to outdated images being reused.
Currently, all containerized workloads have their image pull policy set to
IfNotPresent.This change was originally made to facilitate the e2e testing setup (where the image cannot be pulled due to a lack of a registry).
However, this default comes with a huge downside: repeatedly re-submitting a workload won't work (meaning, the old image will be re-used), unless the node image cache is cleaned manually (which first requires removing any stopped containers using the old workload image).
A more sensible solution would be to expose this image pull policy as part of the job submission options so that users can choose for themselves. IMO,
Alwayswould be the safest default to prevent any surprises due to outdated images being reused.