search

aws / efs-utils

Utilities for Amazon Elastic File System (EFS)
MIT License
294 stars 188 forks source link

Failed to initialize TLS tunnel for [AWS EFS ID] on Ubuntu 18 FIPS kernel #65

Closed D4V3M0NK closed 4 years ago

D4V3M0NK commented 4 years ago

Am running on Ubuntu 18.04.4 server with the certified FIPS 140-2 kernel, have created a script to automate mounting to my encrypted EFS:

efsHost=AWS_FS_ID
sudo apt install -y git make binutils jq
git clone https://github.com/aws/efs-utils
cd efs-utils
./build-deb.sh
sudo apt-get -y install ./build/amazon-efs-utils*deb

# mount EFS
sudo mkdir -p /mnt/efs
sudo mount -t efs -o tls ${efsHost}:/ /mnt/efs

The response that I get is Failed to initialize TLS tunnel for AWS_FS_ID.

Could the FIPS 140-2 kernel be causing the issue? I did not compile stunnel separately as this is obviously quite a recent version of the OS.

If it helps:

$ uname -r
4.15.0-1011-fips

$ openssl version
OpenSSL 1.1.1  11 Sep 2018

... and if I attach to my EFS at the point of spinning up the instance (using the AWS console) it connects just fine (although I don't believe that uses transport encryption which I do require), which leads me to think that it's not the kernel, but thought I'd mention it all the same...

From the /var/log/amazon/efs/mount.logfile:

2020-06-03 22:19:16,428 - ERROR - Failed to start TLS tunnel (errno=1). stdout="b''" stderr="b'[ ] Clients allowed=500\n[.] stunnel 5.44 on x86_64-pc-linux-gnu platform\n[.] Compiled with OpenS
SL 1.1.0g  2 Nov 2017\n[.] Running  with OpenSSL 1.1.1  11 Sep 2018\n[.] Update OpenSSL shared libraries or rebuild stunnel\n[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP
,PSK,SNI Auth:LIBWRAP\n[ ] errno: (*__errno_location ())\n[.] Reading configuration from file /run/efs/stunnel-config.fs-d0a344d5.mnt.efs.20403\n[.] UTF-8 byte order mark not detected\n[.] FIPS
 mode disabled\n[ ] Compression disabled\n[ ] PRNG seeded successfully\n[ ] Initializing service [efs]\n[!] SSL_CTX_new: 140A90F2: error:140A90F2:SSL routines:SSL_CTX_new:unable to load ssl3 md
5 routines\n[!] Service [efs]: Failed to initialize TLS context'"
Cappuccinuo commented 4 years ago

Hey @D4V3M0NK ,

Thanks for the report.

Is this issue persistent? Can you turn on the stunnel log debug in efs-utils.conf(usually under /etc/amazon/efs) and share your stunnel log(usually under /var/log/amazon/efs/fs-****.stunnel.log)?

Thanks.

D4V3M0NK commented 4 years ago

Yes consistently failing.

stunnel_debug_enabled = true
stunnel_logs_file = /var/log/amazon/efs/{fs_id}.stunnel.log

Then running sudo mount -t efs -o tls fs-FS_ID:/ /mnt/efs returns the same failure to initialize the TLS tunnel, but no log files are generated in /var/log/amazon/efs/ as shown below:

~$ sudo ls -l /var/log/amazon/efs/fs*
ls: cannot access '/var/log/amazon/efs/fs*': No such file or directory

Sorry I appreciate that that's not a lot of help...

D4V3M0NK commented 4 years ago

If it helps:

~$ stunnel -version
stunnel 5.44 on x86_64-pc-linux-gnu platform
Compiled with OpenSSL 1.1.0g  2 Nov 2017
Running  with OpenSSL 1.1.1  11 Sep 2018
Update OpenSSL shared libraries or rebuild stunnel
Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP

Global options:
pid                    = /var/run/stunnel4.pid
RNDbytes               = 64
RNDfile                = /dev/urandom
RNDoverwrite           = yes

Service-level options:
ciphers                = FIPS (with "fips = yes")
ciphers                = HIGH:!DH:!aNULL:!SSLv2 (with "fips = no")
curve                  = prime256v1
debug                  = daemon.notice
logId                  = sequential
options                = NO_SSLv2
options                = NO_SSLv3
sessionCacheSize       = 1000
sessionCacheTimeout    = 300 seconds
stack                  = 65536 bytes
TIMEOUTbusy            = 300 seconds
TIMEOUTclose           = 60 seconds
TIMEOUTconnect         = 10 seconds
TIMEOUTidle            = 43200 seconds
verify                 = none

I don't much like the Update OpenSSL shared libraries or rebuild stunnel reference here...

D4V3M0NK commented 4 years ago

Again, if it helps, from /var/log/amazon/efs/mount.log:

2020-06-03 23:42:05,467 - INFO - version=1.25-3 options={'rw': None, 'tls': None}
2020-06-03 23:42:05,600 - INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.fs-FSID.mnt.efs.20054"
2020-06-03 23:42:05,602 - INFO - Started TLS tunnel, pid: 13693
2020-06-03 23:42:05,606 - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ /mnt/efs -o rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20054"
2020-06-03 23:42:06,108 - ERROR - Failed to start TLS tunnel (errno=1). stdout="b''" stderr="b'[ ] Clients allowed=500\n[.] stunnel 5.44 on x86_64-pc-linux-gnu platform\n[.] Compiled with OpenS
SL 1.1.0g  2 Nov 2017\n[.] Running  with OpenSSL 1.1.1  11 Sep 2018\n[.] Update OpenSSL shared libraries or rebuild stunnel\n[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP
,PSK,SNI Auth:LIBWRAP\n[ ] errno: (*__errno_location ())\n[.] Reading configuration from file /run/efs/stunnel-config.fs-FSID.mnt.efs.20054\n[.] UTF-8 byte order mark not detected\n[.] FIPS
 mode disabled\n[ ] Compression disabled\n[ ] PRNG seeded successfully\n[ ] Initializing service [efs]\n[!] SSL_CTX_new: 140A90F2: error:140A90F2:SSL routines:SSL_CTX_new:unable to load ssl3 md
5 routines\n[!] Service [efs]: Failed to initialize TLS context'"
Cappuccinuo commented 4 years ago

If it helps:

~$ stunnel -version
stunnel 5.44 on x86_64-pc-linux-gnu platform
Compiled with OpenSSL 1.1.0g  2 Nov 2017
Running  with OpenSSL 1.1.1  11 Sep 2018
Update OpenSSL shared libraries or rebuild stunnel
Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP

Global options:
pid                    = /var/run/stunnel4.pid
RNDbytes               = 64
RNDfile                = /dev/urandom
RNDoverwrite           = yes

Service-level options:
ciphers                = FIPS (with "fips = yes")
ciphers                = HIGH:!DH:!aNULL:!SSLv2 (with "fips = no")
curve                  = prime256v1
debug                  = daemon.notice
logId                  = sequential
options                = NO_SSLv2
options                = NO_SSLv3
sessionCacheSize       = 1000
sessionCacheTimeout    = 300 seconds
stack                  = 65536 bytes
TIMEOUTbusy            = 300 seconds
TIMEOUTclose           = 60 seconds
TIMEOUTconnect         = 10 seconds
TIMEOUTidle            = 43200 seconds
verify                 = none

I don't much like the Update OpenSSL shared libraries or rebuild stunnel reference here...

This command has the exactly same result as my working ubuntu18. So I think that is fine for that Update OpenSSL shared libraries or rebuild stunnel.

I think the root cause is error:140A90F2:SSL routines:SSL_CTX_new:unable to load ssl3 md 5 routines, have you tried with remove stunnel and re-install it again?

Cappuccinuo commented 4 years ago

Yes consistently failing.

stunnel_debug_enabled = true
stunnel_logs_file = /var/log/amazon/efs/{fs_id}.stunnel.log

Then running sudo mount -t efs -o tls fs-FS_ID:/ /mnt/efs returns the same failure to initialize the TLS tunnel, but no log files are generated in /var/log/amazon/efs/ as shown below:

~$ sudo ls -l /var/log/amazon/efs/fs*
ls: cannot access '/var/log/amazon/efs/fs*': No such file or directory

Sorry I appreciate that that's not a lot of help...

Can you run the mount as the root bash, e.g. sudo su. The log should exist /var/log/amazon/efs, can you ls -l /var/log/amazon/efs after the mount?

D4V3M0NK commented 4 years ago

No difference running as root bash - no log files created.

# ls -l /var/log/amazon/efs/fs*
ls: cannot access '/var/log/amazon/efs/fs*': No such file or directory

Removed stunnel (which also removed efs-utils) then immediately tried the mount and got a different error, but I'm guessing that's because the EFS type is no longer available? 

# sudo mount -t efs -o tls fs-FS_ID:/ efs
mount: /home/ubuntu/efs: special device fs-FS_ID:/ does not exist.

So I decided to build efs-utils again, just in case the build saw that I already had stunnel and therefore wouldn't try building that component again, but same result and no logs .

Potentially a daft question, but am I supposed to have a /etc/stunnel/stunnel.conf?

# stunnel
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g  2 Nov 2017
[.] Running  with OpenSSL 1.1.1  11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "/etc/stunnel/stunnel.conf"
[!] realpath: No such file or directory (2)
D4V3M0NK commented 4 years ago

Unless I'm going down a rabbit hole here, from mount.log

# more /var/log/amazon/efs/mount.log                                  
2020-06-04 02:11:03,608 - INFO - version=1.25-3 options={'rw': None, 'tls': None}
2020-06-04 02:11:05,292 - INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20119"
2020-06-04 02:11:05,296 - INFO - Started TLS tunnel, pid: 4112
2020-06-04 02:11:05,307 - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ efs -o rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20119"
2020-06-04 02:11:05,315 - ERROR - Failed to mount fs-FS_ID.efs.us-west-2.amazonaws.com at efs: returncode=32, stderr="b'mount.nfs4: mount point efs does not exist'"
2020-06-04 02:11:13,946 - INFO - version=1.25-3 options={'rw': None, 'tls': None}
2020-06-04 02:11:14,050 - INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20184"
2020-06-04 02:11:14,052 - INFO - Started TLS tunnel, pid: 4131
2020-06-04 02:11:14,059 - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ /home/ubuntu/efs -o rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20184"
2020-06-04 02:11:14,560 - ERROR - Failed to start TLS tunnel (errno=1). stdout="b''" stderr="b'[ ] Clients allowed=500\n[.] stunnel 5.44 on x86_64-pc-linux-gnu platform\n[.] Compiled with OpenS
SL 1.1.0g  2 Nov 2017\n[.] Running  with OpenSSL 1.1.1  11 Sep 2018\n[.] Update OpenSSL shared libraries or rebuild stunnel\n[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP
,PSK,SNI Auth:LIBWRAP\n[ ] errno: (*__errno_location ())\n[.] Reading configuration from file /run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20184\n[.] UTF-8 byte order mark not detected\n
[.] FIPS mode disabled\n[ ] Compression disabled\n[ ] PRNG seeded successfully\n[ ] Initializing service [efs]\n[!] SSL_CTX_new: 140A90F2: error:140A90F2:SSL routines:SSL_CTX_new:unable to load
 ssl3 md5 routines\n[!] Service [efs]: Failed to initialize TLS context'"
2020-06-04 02:18:36,948 - INFO - version=1.25-3 options={'rw': None, 'tls': None}
2020-06-04 02:18:37,109 - INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs-utils.efs.20311"
2020-06-04 02:18:37,114 - INFO - Started TLS tunnel, pid: 4777
2020-06-04 02:18:37,127 - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ efs -o rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20311"
2020-06-04 02:18:37,136 - ERROR - Failed to mount fs-FS_ID.efs.us-west-2.amazonaws.com at efs: returncode=32, stderr="b'mount.nfs4: mount point efs does not exist'"
2020-06-04 02:18:49,319 - INFO - version=1.25-3 options={'rw': None, 'tls': None}
2020-06-04 02:18:49,425 - INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20131"
2020-06-04 02:18:49,427 - INFO - Started TLS tunnel, pid: 4798
2020-06-04 02:18:49,431 - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ /home/ubuntu/efs -o rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20131"
2020-06-04 02:18:49,933 - ERROR - Failed to start TLS tunnel (errno=1). stdout="b''" stderr="b'[ ] Clients allowed=500\n[.] stunnel 5.44 on x86_64-pc-linux-gnu platform\n[.] Compiled with OpenS
SL 1.1.0g  2 Nov 2017\n[.] Running  with OpenSSL 1.1.1  11 Sep 2018\n[.] Update OpenSSL shared libraries or rebuild stunnel\n[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP
,PSK,SNI Auth:LIBWRAP\n[ ] errno: (*__errno_location ())\n[.] Reading configuration from file /run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20131\n[.] UTF-8 byte order mark not detected\n
[.] FIPS mode disabled\n[ ] Compression disabled\n[ ] PRNG seeded successfully\n[ ] Initializing service [efs]\n[!] SSL_CTX_new: 140A90F2: error:140A90F2:SSL routines:SSL_CTX_new:unable to load
 ssl3 md5 routines\n[!] Service [efs]: Failed to initialize TLS context'"

and then when I review /run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20131 I get this:

# more /run/efs/stunnel-config.fs-FS_ID.home.ubuntu.efs.20131
fips = no
foreground = yes
socket = l:SO_REUSEADDR=yes
socket = a:SO_BINDTODEVICE=lo
debug = debug
output = /var/log/amazon/efs/fs-FS_ID.stunnel.log
[efs]
client = yes
accept = 127.0.0.1:20131
connect = fs-FS_ID.efs.us-west-2.amazonaws.com:2049
sslVersion = TLSv1.2
renegotiation = no
TIMEOUTbusy = 20
TIMEOUTclose = 0
TIMEOUTidle = 70
delay = yes
verify = 2
CAfile = /etc/amazon/efs/efs-utils.crt
cert = /var/run/efs/fs-FS_ID.home.ubuntu.efs.20131+/certificate.pem
key = /etc/amazon/efs/privateKey.pem
checkHost = fs-FS_ID.efs.us-west-2.amazonaws.com
libwrap = no

And the first thing I notice, is that it's not running in FIPS mode.

Cappuccinuo commented 4 years ago

The stunnel config seems fine, and the /etc/stunnel/stunnel.conf is not needed.

There is no stunnel log, which is very weird, since in your stunnel config, there is output = /var/log/amazon/efs/fs-FS_ID.stunnel.log, what's your permission of folder /var/log/amazon/efs, is that writable?

D4V3M0NK commented 4 years ago 
# ls -l /var/log/amazon/
total 8
drwxr-xr-x 2 root root 4096 Jun  4 02:11 efs
drwxr--r-x 2 root root 4096 May 29 04:17 ssm

As root, I'm good to go

Cappuccinuo commented 4 years ago

Did you encounter the same issue using other kernel on Ubuntu18?

D4V3M0NK commented 4 years ago

To be honest, I've been 100% focussed on getting this project up and running and all my hosts have to be FIPS compliant, so that's all I'm working with. I can try with a non-FIPS kernel for comparison?

Cappuccinuo commented 4 years ago

I already tried with non-FIPS kernel and works fine, so that might not be worth to do.

I will try to use the kernel 4.15.0-1011-fips you use. Is there any special steps for me to modify the other kernel on Ubuntu18 to this one?

D4V3M0NK commented 4 years ago

In order for you to use the certified kernel, you'll need an Ubuntu support contract and subsequent access to the FIPS private PPA. I have a script that then does everything for you, within an AWS instance. I'd be more than happy to provide that script, it takes 4-5 mins to run.

(And I will probably try a non-FIPS kernel, just to ensure that I've not set up something odd with my EFS mount points.)

Cappuccinuo commented 4 years ago

Thanks. Have you try with https://github.com/aws/efs-utils/blob/master/src/mount_efs/__init__.py#L177 changing FIPS to yes?

D4V3M0NK commented 4 years ago

Yes! Thanks @Cappuccinuo that totally did it! Thank you !!

D4V3M0NK commented 4 years ago

I have the following scripted, hopefully it helps:

aws - efs-utils

git clone https://github.com/aws/efs-utils

reconfigure stunnel to use fips mode

cd efs-utils sed -i -E "s/'fips': 'no'/'fips': 'yes'/" ./src/mount_efs/init.py ./build-deb.sh sudo apt install -y ./build/amazon-efs-utils*deb cd ..

On Sat, 2020-10-03 at 01:41 -0700, Sergey Kharitonov wrote:

Hello @D4V3M0NK @Cappuccinuo I got the same problem, could you please clarify where option fips = yes must be placed? In my case in each time when trying to mount efs generates new config in /var/run/efs/stunnel-config.fs-*.efs.20291 where fips option set to no, if change file and manually start stunnel with changed file all works fine

/var/run/efs/stunnel-config.fs-*****.efs.20291

/var/run/efs/stunnel-config.fs-*****.efs.20295

/var/run/efs/stunnel-config.fs-*****.efs.20424

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/aws/efs-utils/issues/65#issuecomment-703069772", "url": "https://github.com/aws/efs-utils/issues/65#issuecomment-703069772", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

philipdumont commented 2 years ago

This is great, that there's a way to force it to work. But could someone please explain, why do I have to force it to work? Why is the 'fips' key value hard-coded to 'no'? There are clearly some users for whom that is not desirable. Therefore, why not make it a command-line option? Making me get what I need by editing a packaged executable (and thereby making "rpm -qV" complain) seems less than ideal.

philipdumont commented 2 years ago

Okay, yeah, so the instructions don't actually say to edit the packaged file. They say to edit the file in your source repo clone, package it yourself, install your custom-built package. Still, not ideal.

Especially since the folks who require FIPS and encryption-in-transit and all that rot are also the ones most likely to be less-than-pleased about needing to use non-stock packages.

Cappuccinuo commented 2 years ago

Hey @philipdumont , what version of efs-utils are you using? Can you pull the latest version, and follow https://github.com/aws/efs-utils#enabling-fips-mode this section to enable FIPS mode? The only thing needed here is modify one line of efs-utils configuration file.

philipdumont commented 2 years ago

Well, that is cleaner.

I was indeed using the latest version -- the latest available in the RH repos, anyway: amazon-efs-utils-1.33.3-1.el7_9.noarch. I just didn't know about the config file thing.

BTW, the sed command presented there wasn't quite right -- at least, not for my version of the config file. But this did the job:

sudo sed -i "s/stunnel_fips_enabled = false/stunnel_fips_enabled = true/" /etc/amazon/efs/efs-utils.conf

Cappuccinuo commented 2 years ago

There could be some delay on the repo sync, the latest one 1.34.1 in Github repo changed the config item name, since we also enable botocore to use fips endpoint: https://github.com/aws/efs-utils/blob/master/dist/efs-utils.conf#L33

philipdumont commented 2 years ago

Oh, yeah, RH is usually a little behind the "bleeding edge".

That's good to know. We should make our sed command work for either of those possible keywords, so it will still work when RH catches up. Thanks.