Open 1337andre opened 2 years ago
As an alternative approach, perhaps kube-fledged could watch for a certain pod annotation to automatically add the requested image to the cache, and subscribe the node to the image.
This functionality could work similar to cert-manager, which watches for a cert-manager.io/cluster-issuer annotation on ingresses to create a Certificate Resource, rather than requiring the user to create one manually.
Automatically determining which images to be cached is in the roadmap of kube-fledged. Would like to see more elaborative details on how best this could be implemented.
Hey @senthilrch ,
What do you think about controlling the same through annotations?
kubefledged.io/enable: "true"
.true
with another annotation like kubefledged.io/node-selector: "somekey=somevalue"
then the controller only creates cache on nodes matching the label. Also, if this annotation is present, then there is no need to check if kubefledged.io/enable
is set or not. A Pod could have multiple init containers and app containers. We should have a means to specify whether to cache all the container images or only a subset of container images.
Right - maybe we can control the same using another annotation like kubefledged.io/pod-images: "image-*,image-*"
. If the annotation is present, only pull the mentioned images (with a wildcard), else pull all images in the pod.
Given that a pod requires all of its images to be present to start up, I think it would actually be a pretty rare occurrence to only want a subset of images on a pod to be pulled. Maybe that level of detail could come later?
@senthilrch @1337andre and whoever is interested:
Until this feature is implemented, I created a wrapper chart for kube-fledged which manages ImageCaches automatically with kubernetes events. It's not yet perfect but it gets the job done:
@1337andre @tculp @vikas027 :
I propose to create a new CRD to allow harvesting images for the cache from Kubernetes objects. This allows a less-intrusive (resources needs not be labelled/annotated) and highly flexible way of declaring the "sources" for harvesting images. Even images from customer resources (e.g. ArgoWF, SeldonDeployment etc.) can be supported with this design approach:-
apiVersion: kubefledged.io/v1alpha1
kind: ImageCacheBuilder
metadata:
# Name of the image cache. A cluster can have multiple image cache objects
name: imagecache2
namespace: kube-fledged
# The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
labels:
app: kubefledged
kubefledged: imagecache
spec:
- apiGroups:
- "apps"
resources:
- statefulsets
namespaceSelector:
matchNames:
- sock-shop
selector:
matchLabels:
name: front-end
- apiGroups:
- "apps"
resources:
- deployments
namespaceSelector:
matchNames:
- sock-shop
selector:
matchLabels:
name: front-end
Harvesting images from Deployments and Statefulsets will be supported in v0.11.0. i.e. Pods have to be created using these controllers. Images won't be harvested from Pods.
Hi, thanks for great project.
We have separated our processes by namespaces. e.g. monitoring, logging, metering, velero... each namespace have it's own CR imagecache and its own pullSecret.
Would it be possible to add a flag to take/cache all images in namespace? This way we don't need to maintain image in cache list and in deployment itself.
greetings Andre