search

kubernetes-sigs / aws-efs-csi-driver

CSI Driver for Amazon EFS https://aws.amazon.com/efs/
Apache License 2.0
723 stars 554 forks source link

Flexibility to increase the EFS TLS Ports via /etc/amazon/efs/efs-utils.conf #1398

Open rshiva777 opened 4 months ago

rshiva777 commented 4 months ago

/kind bug

What happened? When i run pods with multiple EFS volume mounts, some pods are into pending state due to EFS ports are fully occupied (20049 - 20449)

What you expected to happen? I expect an way to increase the EFS TLS ports from (20049 - 20449) as per the requirement

How to reproduce it (as minimally and precisely as possible)?

Create a deployment with 10 EFS volume mounts, scale the replica to 50 Anything else we need to know?:

I tried adding a configmap with efs-utils.conf file with updated ports, then mounted it to the csi-node daemonset. Post that i see the below error on daemonset

F0701 15:16:18.687401 1 main.go:65] cannot create config file /etc/amazon/efs for efs-utils. Error: open /etc/amazon/efs/efs-utils.conf: read-only file system

Seems, efs-csi is trying to update the efs-utils.conf file mounted via configmap and getting failed.

Environment

Please also attach debug logs to help us better diagnose

Defaulted container "efs-plugin" out of: efs-plugin, csi-driver-registrar, liveness-probe I0701 15:16:18.674262 1 config_dir.go:56] Symlink or directory exists at '/etc/amazon/efs', no need to create one I0701 15:16:18.674972 1 metadata.go:65] getting MetadataService... I0701 15:16:18.676846 1 metadata.go:70] retrieving metadata from EC2 metadata service I0701 15:16:18.685570 1 driver.go:150] Did not find any input tags. I0701 15:16:18.686302 1 driver.go:116] Registering Node Server I0701 15:16:18.686420 1 driver.go:118] Registering Controller Server I0701 15:16:18.686512 1 driver.go:121] Starting efs-utils watchdog I0701 15:16:18.686885 1 efs_watch_dog.go:221] Skip copying /etc/amazon/efs/efs-utils.conf since it exists already I0701 15:16:18.686964 1 efs_watch_dog.go:216] Copying /etc/amazon/efs/efs-utils.crt since it doesn't exist F0701 15:16:18.687401 1 main.go:65] cannot create config file /etc/amazon/efs for efs-utils. Error: open /etc/amazon/efs/efs-utils.conf: read-only file system

daro1337 commented 2 months ago

I had the same error. In my case, I wanted to change the log level in /etc/amazon/efs/efs-utils.conf, but I am not able to apply my configuration.

avanish23 commented 1 month ago

Hi @rshiva777 Where are you getting the 20049 - 20449 port range numbers; officially the ports allowed are 20049 to 21049. Please share the error you see when the port range is exhausted.

Coming to the questions as to why the port range did not get updated even when you had configured the config maps is because the port range is not configurable and we do not pick the values up from the env variables; as of date only AWS_DEFAULT_REGION and FIPS_ENABLED are configurable and are picked up from the env variables.

@daro1337 even in your case since the log level is currently not configurable you could not set them to DEBUG. However please refer to the following doc to enable DEBUG logs.

Given this maybe @rshiva777 even you can change the port range by shelling into the driver pod. Please try this and let us know if it works.

Tagging @mskanth972 to confirm the same; can we have a configurable port range and log level via a config map?

avanish23 commented 1 month ago

/assign

mskanth972 commented 1 month ago

@avanish23, you're right. However, we recently expanded the driver’s port range from 400 to 1000. I suggest trying the latest version, v2.0.8. However, if you're using addons, note that v2.0.8 hasn't been made public yet, so you'll need to use v2.0.7 instead.

The reason you're not seeing any changes after updating the config file is that the EFS CSI Driver uses a separate config file located in watchdog.go. So, even if you modify the EFS Utils config file, it defaults back to the values specified in the watchdog file.

You can refer to this PR for more details: #1402.