search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
55 stars 707 forks source link

Having Large Plugins in a Custom Image Causes Pods to be Evicted #6542

Open zeezoich opened 1 year ago

zeezoich commented 1 year ago

Bug Report

What did you do? I built a custom Elasticsearch image version 7.17.2. It has plugins installed already. These are large plugins >2 GB. What did you expect to see? I expect the elasticsearch pod to start normally and the cluster to be operational.

What did you see instead? Under which circumstances? Whenever I deploy the cluster using the operator, the elasticsearch pod gets evicted due to exceeding the default ephemeral storage limit. This is tried with operator version 2.0.0 and 2.6.1.

Environment Kubernetes v. 1.24

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-08-17T18:47:37Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/amd64"}

Additional Info The plugins get copied from one directory to another (volume to another) by an init-container. I wonder why that's happening, if the custom image has the plugins I need installed, why do need the copying, surely there is a reason and I am curious as to why. This is specifically what I am referring to https://github.com/elastic/cloud-on-k8s/blob/main/pkg/controller/elasticsearch/initcontainer/prepare_fs_script.go#L116

Thank you.

zeezoich commented 1 year ago

Likely a duplicate of #6406 and could be closed.

thbkrkr commented 1 year ago

Likely a duplicate of https://github.com/elastic/cloud-on-k8s/issues/6406 and could be closed.

It doesn't seem to be a duplicate. Your pod is evicted due to lack of storage. In #6406, pod is OOMKilled due to lack of memory.

Can you share the plugin you are installing?

zeezoich commented 1 year ago

@thbkrkr it's RBL-Elasticsearch plugin. Take a look, https://support.rosette.com/hc/en-us/articles/360052972331-Getting-Started

johnswarbrick-napier commented 1 year ago

Hi @zeezoich I have the same issue with the RBL plugin. Did you find a solution?

zeezoich commented 1 year ago

@johnswarbrick-napier I outlined a workaround here https://github.com/elastic/cloud-on-k8s/issues/6406#issuecomment-1473085117 the idea is to use a non-ephemeral-storage volume and alter the init-container to mount the volume where the plugins will be copied.

johnswarbrick-napier commented 1 year ago

Thanks @zeezoich I'll check that out. I'm still curious why it copies the plugins between directories at all though - seems like an unnecessary extra overhead at startup?