search

aws / efs-utils

Utilities for Amazon Elastic File System (EFS)
MIT License
301 stars 192 forks source link

High CPU usage after upgrading from 1.31 to 2.1 due to efs-proxy (even without tls mount option) #257

Open DoobleD opened 1 week ago

DoobleD commented 1 week ago

Hi EFS team,

We recently upgraded our efs-utils from 1.31 to 2.1 and re-mounted our fairly busy EFS. Since then, we observe a new efs-proxy process that's taking a lot of CPU:

/usr/bin/efs-proxy /var/run/efs/stunnel-config.fs-XXXXX.YYYY.20385

On a 4 vcpu / 32 GB EC2 instance, this uses about 20% CPU most of the time we look at it, according to glances.

Since efs-proxy is there to replace stunnel for tls encryption in transit (or at least that's our understanding), we tried re-mounting without the tls option. But efs-proxy is still running even in that case, and still taking as much CPU.

$ uname -a
Linux ip-172-31-25-124 5.15.0-1070-aws #76~20.04.1-Ubuntu SMP Mon Sep 2 12:20:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
$ mount | grep '/YYYY'
127.0.0.1:/ on /YYYY type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,port=20385,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1,_netdev)

Any ideas on how to resolve this? Happy to get in touch with the team for a more advanced investigation if needed.

anthotse commented 4 days ago

In efs-utils 2.0 and greater, efs-proxy replaces stunnel to perform TLS encryption and to enable higher per-client throughput (up to 1,500 mebibytes per second) when mounting to a file system using the Elastic Throughput performance mode.

If you require TLS encryption and this issue is related to differences between stunnel and efs-proxy, you can mount with the "stunnel" included as a mount option.

DoobleD commented 4 days ago

Thank you for your reply @anthotse! Good to know that we could revert to using stunnel.

We're not entirely sure that we require TLS encryption since we mount EFS in the VPC only. What's surprising is that efs-proxy is still running even when the tls option is omitted when mounting the EFS.

Since efs-proxy is supposed to replace stunnel and that stunnel is used only for the tls option (I think), shouldn't efs-proxy not run when mounting without the tls option?

anthotse commented 2 days ago

efs-proxy is run when mounting without the TLS option to enable higher per-client throughput (up to 1,500 mebibytes per second) when mounting to a file system using the Elastic Throughput performance mode.