Improve build performance via caching

[lehrig]

/kind feature

Why you need this feature: Minimizing CI build times is always good! My team estimates that enabling caching can lower building times of Kubeflow components by 30-40% on average.

Describe the solution you'd like: Enable caching in GHA: if workflows are updated accordingly, builds can reuse previously build container layers (if they haven't been changed). An internal image registry has to be configured for this to be possible. Optimally, Dockerfiles are also optimized for leveraging caching, e.g., by avoiding IF/ELSE constructs in RUN commands and by moving often-modified parts to the end of the Dockerfile.

Anything else you would like to add: This will benefit all components and all architectures, even though this is especially critical for emulated builds (see https://github.com/kubeflow/community/issues/781).

[kimwnasptd]

This is a great proposal! Do you have any other OSS project that is utilizing it so that we could take a look for inspirations on how to implement it in Kubeflow as well?

[andreyvelich]

Let's discuss the cross-component ideas in the community repo. /transfer community

[thesuperzapper]

For reference, we already do this in most of the kubeflow/kubeflow builds.