Closed g00ntar closed 3 years ago
It is possible to use the config
parameter from the values.yaml in order to provide a custom ES configuration, see
## Customize elasticsearch configuration
## ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html
##
# config:
which it's going to create a ConfigMap with the data provided in this section and mounted to /opt/bitnami/elasticsearch/config/elasticsearch.yml
. You should be able to set any parameter supported by ES in this config file
I looking to change this parameter because this allow to run ElasticSearch pod as unprivileged and allow to install this helm on GKE Autopilot. After your answer I created myvalues.yml
with content
sysctlImage:
enabled: true
config:
index.store.type: niofs
node.store.allow_mmap: false
and after running helm install -f myvars.yml test-es-1 bitnami/elasticsearch
container dies in pod with error complaining about Invalid kernel settings. Elasticsearch requires at least: vm.max_map_count = 262144
which should not be a case.
Helm chart provided by ElasticSearch itself with values
sysctlInitContainer:
enabled: false
esConfig:
elasticsearch.yml: |
index.store.type: niofs
node.store.allow_mmap: false
works normally - it skips checking value of vm.max_map_count
Currently, Elasticsearch requires some changes in the kernel of the host machine to work as expected. If those values are not set in the underlying operating system, the ES containers fail to boot with ERROR messages. Here you can find more info about the specific case of Virtual Memory.
The Bitnami Helm Chart try to solve this issue by adding an initContainer to change those settings in the Kernel by running: sysctl -w vm.max_map_count=262144 && sysctl -w fs.file-max=65536
. This initContainer should be executed as a privileged container since it is needed to change the kernel settings. This container is enabled by default, but it can be disabled by using sysctlImage.enabled=false
, for example, if the kernel already has the proper settings.
Yes, you are right and I'm aware of this need, but... After https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-virtual-memory.html
Note that this requires the ability to run privileged containers, which is likely not the case on many secure clusters.
and there (on the same website) are solutions/workaround to make this works without need to change kernel settings.
To run elesticsearch without need max_map_count=262144
setting (and at the same time drop depency of privileged initialContainer
) config of elesticsearch must contain index.store.type: niofs
or node.store.allow_mmap: false
(and thats why I provided working config for elasticsearch helm chart).
I understand that maybe bitnami are not interested to make changes for this sepecyfic use case (GKE) but still changes in config are not respected by the chart.
I'm not native speaker and maybe I'm not quite clear, so to help understand the case I found similar issue on elasticsearch github #32267
The privileged initContainer is enabled by default, in order to disable it, you need to set sysctlImage.enabled=false
when installing the chart. Then, you should be able to set the desired parameters in the configuration file, something like
sysctlImage:
enabled: false
config:
index:
store:
type: niofs
node:
store:
allow_mmap: false
In that case, are you able to see the proper configuration in the config file (elasticsearch.yml
)? You can check it by accessing the pod (if it is up and running) or taking a look at the ConfigMap with kubectl
.
What is the exact error you are seen in the logs when deploying the chart with the above configuration?
I'm little ambarased because I made mistake in config in my second post, of course there should be sysctlImage.enabled=false
and I tried this from begining.
So below steps to reproduce on GKE with some output - helm chart elastic/elasticsearch with same config works on GKE
cat <<EOF > myvalues.yml
sysctlImage:
enabled: false
config:
index.store.type: niofs
node.store.allow_mmap: false
EOF
helm install -f myvalues.yml bitnami-es bitnami/elasticsearch
kubectl get pods
NAME READY STATUS RESTARTS AGE
bitnami-es-elasticsearch-coordinating-only-85cd75d5b4-ftptg 0/1 CrashLoopBackOff 3 105s
bitnami-es-elasticsearch-coordinating-only-85cd75d5b4-lt2bj 0/1 ContainerCreating 0 105s
bitnami-es-elasticsearch-data-0 0/1 Pending 0 105s
bitnami-es-elasticsearch-data-1 0/1 ContainerCreating 0 104s
bitnami-es-elasticsearch-master-0 0/1 ContainerCreating 0 105s
bitnami-es-elasticsearch-master-1 0/1 ContainerCreating 0 104s
kubectl logs bitnami-es-elasticsearch-coordinating-only-85cd75d5b4-ftptg
elasticsearch 13:58:00.84
elasticsearch 13:58:00.90 Welcome to the Bitnami elasticsearch container
elasticsearch 13:58:00.90 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch
elasticsearch 13:58:00.90 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues
elasticsearch 13:58:00.91
elasticsearch 13:58:00.91 INFO ==> ** Starting Elasticsearch setup **
elasticsearch 13:58:01.02 ERROR ==> Invalid kernel settings. Elasticsearch requires at least: vm.max_map_count = 262144
Thanks for the detailed explanation, I was taking a look and found where the issue comes from.
During the bitnami/elasticsearch
container image initialization, there are different scripts executed, one of those scripts is setup.sh, in this script there is the following function (link):
# Ensure kernel settings are valid
elasticsearch_validate_kernel
This function is defined at libelasticsearch.sh (link):
########################
# Validate kernel settings
# Arguments:
# None
# Returns:
# None
#########################
elasticsearch_validate_kernel() {
# Auxiliary functions
validate_sysctl_key() {
local key="${1:?key is missing}"
local value="${2:?value is missing}"
local current_value
current_value="$(sysctl -n "$key")"
if [[ "$current_value" -lt "$value" ]]; then
error "Invalid kernel settings. Elasticsearch requires at least: $key = $value"
exit 1
fi
}
debug "Validating Kernel settings..."
validate_sysctl_key "vm.max_map_count" 262144
validate_sysctl_key "fs.file-max" 65536
}
This validation was implemented in the past because we were having several support cases related to that and probably modifying the storage type was not an option. I just created an internal task to investigate if we can improve the current container logic to support this use case.
In the meantime, as a workaround, you can build your own image and use it in the Helm chart:
$ git clone https://github.com/bitnami/bitnami-docker-elasticsearch.git
$ cd bitnami-docker-elasticsearch/7/debian-10
$ vim rootfs/opt/bitnami/scripts/elasticsearch/setup.sh # Removing the elasticsearch_validate_kernel line
$ docker build -t myuser/custom-es:latest .
$ docker push myuser/custom-es:latest
image:
registry: docker.io
repository: myuser/custom-es
tag: latest
sysctlImage:
enabled: false
config:
index:
store:
type: niofs
node:
store:
allow_mmap: false
Following the previous approach, you should be able to deploy ES in an unprivileged cluster by disabling the initcontainer and using a custom config file, thanks to the removal of the validation in your custom image.
I'm setting the "on-hold" label to prevent the stale bot from closing the issue while we work on the internal task to investigate what is the real purpose of the validation in the image and if there is a chance of getting rid of it
@g00ntar we are working on some changes in the image to run the kernel settings validation only if node.store.allow_mmap
and/or index.store.type
are not set in the config file. Apart from that, I created the above PR adding more info to the NOTES.txt.
Were you able to solve the issue with the proposed workaround? Basically, the solution we are going to provide is similar. In the proposed workaround the kernel validation was directly removed while in the long-term solution we are going to check the config file to determine if the validation should be executed or not, but the result should be the same
This issue was automatically closed because the changes in the image were already implemented, see tag 7.10.2-debian-10-r56
(https://github.com/bitnami/bitnami-docker-elasticsearch/commit/348956597b96af42be41f8f1d1e07ddb3b4dd648) and the Helm Chart was updated to use this image among other changes related to the docs.
Now using custom config file containing node.store.allow_mmap
and/or index.store.type
in one of the following ways:
sysctlImage:
enabled: false
config:
index:
store:
type: niofs
node:
store:
allow_mmap: false
sysctlImage:
enabled: false
config:
index.store.type: niofs
node.store.allow_mmap: false
the kernel setting validations shouldn't be executed during the container initialization. That means the deployment is not going to fail due to this reason, but then ES should handle properly the configured settings to deploy the Helm Chart in an unprivileged cluster, but this is something not checked/handled by the container image nor the Helm Chart.
Is there way to change index.store.type like on https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-virtual-memory.html in ElasticSearch chart ?