curl under d4m breaks with "sslv3 alert handshake failure"; works fine under linux

victortrac commented 7 years ago

Expected behavior

$ docker run -it --rm php:latest curl -I "https://services.mailup.com"
HTTP/1.1 403 Forbidden
Content-Length: 1233
Content-Type: text/html
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Fri, 13 Jan 2017 19:26:40 GMT

Actual behavior

$ docker run -it --rm php:latest curl -I "https://services.mailup.com"
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

$ cat << EOF > Dockerfile
FROM debian:latest

RUN apt-get update && apt-get -y install curl
EOF

$ docker build -t curl-test .
$ docker run -it --rm curl-test  curl -I "https://services.mailup.com"
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Alpine:

$ cat << EOF > Dockerfile
FROM alpine:latest

RUN apk --update add curl
EOF

$ docker build -t curl-test2 .
$ docker run -it --rm curl-test2  curl -I "https://services.mailup.com"
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Information

This works perfectly fine on my linux machine but fails on 3 different OSX machines, including failing on both docker-for-mac and docker-machine setups. The same curl command works on OSX outside of docker as expected.

Diagnostic ID: 9FEE21CB-F05A-4CB5-84C2-CA33E27039E1

Steps to reproduce the behavior

Run docker run -it --rm php:latest curl -I "https://services.mailup.com" on OSX.

justincormack commented 7 years ago

Hmm, it works for me. Which version of docker for mac do you have installed?

victortrac commented 7 years ago

@justincormack

Version 1.13.0-rc6-beta36 (14969)
Channel: Beta
a158c69c78

$ docker version
Client:
 Version:      1.13.0-rc6
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   2f2d055
 Built:        Wed Jan 11 21:47:55 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      1.13.0-rc6
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   2f2d055
 Built:        Wed Jan 11 21:47:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

$ docker info
Containers: 64
 Running: 0
 Paused: 0
 Stopped: 64
Images: 137
Server Version: 1.13.0-rc6
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 569
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.41-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 3.854 GiB
Name: moby
ID: GAJN:5ONQ:2NQ6:NXAJ:2VEX:ENFG:SOJN:XD6C:LZJ5:G4ZG:FV25:3CPK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 17
 Goroutines: 28
 System Time: 2017-01-13T19:43:24.107936199Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

OSX 10.12.2 (16C67) on a MacBook Pro (Retina, 15-inch, Mid 2015).

justincormack commented 7 years ago

Hmm. Are you running behind any kind of proxy?

AndyBold commented 7 years ago

Also reproduced on Mac.

I ran the same test using d4m from the stable channel, Version 1.12.6 (14937) on Sierra 10.12.2. Not sitting behind any proxies or doing anything odd with networking.

victortrac commented 7 years ago

@justincormack I have a VPN client installed (pritunl) but it's not connected or running. I also explicitly went to Docker for Mac settings -> Proxies -> checked "No Proxy" (was previously the default of "Use system proxy"), but doesn't change the result.

dsheets commented 7 years ago

This appears to be an OpenSSL cipher negotiation problem that is unrelated to Docker for Mac. I can reproduce the issue in debian:latest with curl -I https://services.mailup.com:

curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

TLS 1.0 and 1.1 seem to be unaffected:

curl -I --tlsv1.1 https://services.mailup.com

curl -I --tlsv1.0 https://services.mailup.com

Also, if I force the cipher list with TLS 1.2 to be only AES256-SHA256 or AES256-SHA, the connection succeeds:

curl -I --tlsv1.2 --ciphers AES256-SHA256 https://services.mailup.com

curl in debian:latest is compiled against OpenSSL/1.0.1t:

# curl --version
curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1t zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API SPNEGO NTLM NTLM_WB SSL libz TLS-SRP

whereas, on macOS, curl is compiled against Apple's SecureTransport. If you have another functioning Linux userland, I suggest investigating which version of OpenSSL (or NSS or GnuTLS) curl has been built against. That may lead you to an issue with either curl or the server at services.mailup.com.

Because this does not appear to be a Docker for Mac issue, I am going to close it as status/0-wont-fix. Good luck in your search for the root cause of the TLS negotiation problem! If you do believe that this issue is still Docker for Mac, please feel free to re-open with an explanation of why you believe Docker for Mac is causing the problem. Thanks!

dsheets commented 7 years ago

Oh, I forgot to mention that openssl s_client -connect services.mailup.com:443 also connects correctly so the problem may in fact lie with curl.

victortrac commented 7 years ago

@dsheets You've seemed to just confirmed the issue..?

The same docker image with the same binary of curl works fine on all linux docker hosts I've tried. This fails on OSX with alpine and curl 7.52.1. See my alpine Dockerfile in my original comment.

This seems to me like something in the networking layer is chopping or mangling packets such that ssv3 handshake is failing on osx. Can you elaborate on why you think this isn't the case?

dsheets commented 7 years ago

@victortrac I've confirmed the issue is not related to Docker for Mac:

On a Linux 4.2.0 machine running docker 1.10.3, I see

$ sudo docker run --rm -it php:latest curl -I https://services.mailup.com
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

We haven't had any other reports about network errors for many months (certainly not in Beta 36). Are you sure that the same image and curl binary work for you on Docker on Linux machines?

All of the evidence I have points to this being an issue with the TLS configuration of https://services.mailup.com or curl or a combination of both.

victortrac commented 7 years ago

@dsheets Some more datapoints for you.

My linux laptop:

$ uname -a
Linux callisto 4.10.0-1-drm-intel-nightly #28 SMP PREEMPT Wed Jan 11 17:48:49 CST 2017 x86_64 GNU/Linux

$ docker --version
Docker version 1.12.6, build 78d18021ec

$ docker inspect php:latest
[
    {
        "Id": "sha256:97c69bd40b40e1a1338409408ceb61b779097cc857037305450ad9a1ea249695",
        "RepoTags": [
            "php:latest"
        ],
        "RepoDigests": [
            "php@sha256:27c3c78c22adcb782372890fc0a790409ef9a6bdfc30e34b69719e38774c09ec"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2017-01-17T19:11:16.191835022Z",
        "Container": "20a77fd84e12917377c53085f8ba6a5fa1dd7bcac2b9d1407ac52533276bc75a",
        "ContainerConfig": {
            "Hostname": "33842653d6db",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PHPIZE_DEPS=autoconf \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c",
                "PHP_INI_DIR=/usr/local/etc/php",
                "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_LDFLAGS=-Wl,-O1 -Wl,--hash-style=both -pie",
                "GPG_KEYS=A917B1ECDA84AEC2B568FED6F50ABC807BD5DCD0",
                "PHP_VERSION=7.1.0",
                "PHP_URL=https://secure.php.net/get/php-7.1.0.tar.xz/from/this/mirror",
                "PHP_ASC_URL=https://secure.php.net/get/php-7.1.0.tar.xz.asc/from/this/mirror",
                "PHP_SHA256=a810b3f29c21407c24caa88f50649320d20ba6892ae1923132598b8a0ca145b6",
                "PHP_MD5=cf36039303c47f493100afea522a8f53"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"php\" \"-a\"]"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:2826284d23a26e9078f99960f834ebe17561a0b8a152d0398446f3d2ef24b3b7",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "docker-php-entrypoint"
            ],
            "OnBuild": [],
            "Labels": {}
        },
        "DockerVersion": "1.12.3",
        "Author": "",
        "Config": {
            "Hostname": "33842653d6db",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PHPIZE_DEPS=autoconf \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c",
                "PHP_INI_DIR=/usr/local/etc/php",
                "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_LDFLAGS=-Wl,-O1 -Wl,--hash-style=both -pie",
                "GPG_KEYS=A917B1ECDA84AEC2B568FED6F50ABC807BD5DCD0",
                "PHP_VERSION=7.1.0",
                "PHP_URL=https://secure.php.net/get/php-7.1.0.tar.xz/from/this/mirror",
                "PHP_ASC_URL=https://secure.php.net/get/php-7.1.0.tar.xz.asc/from/this/mirror",
                "PHP_SHA256=a810b3f29c21407c24caa88f50649320d20ba6892ae1923132598b8a0ca145b6",
                "PHP_MD5=cf36039303c47f493100afea522a8f53"
            ],
            "Cmd": [
                "php",
                "-a"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:2826284d23a26e9078f99960f834ebe17561a0b8a152d0398446f3d2ef24b3b7",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "docker-php-entrypoint"
            ],
            "OnBuild": [],
            "Labels": {}
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 364487897,
        "VirtualSize": 364487897,
        "GraphDriver": {
            "Name": "devicemapper",
            "Data": {
                "DeviceId": "690",
                "DeviceName": "docker-254:2-1310723-756c6d5bf8881cf8cbe1013b1bd7f8505870d61146dc8ac8bdb7df5ae2897e5a",
                "DeviceSize": "10737418240"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c",
                "sha256:f134b0bf5c584dfb300c39e73b61ea5546b225e6a689e0cb44a45185a25fb95e",
                "sha256:8173a4e80171357ef8a763cbc5a2e24353711fede10d72ca98d867a9a17f6f9f",
                "sha256:b484971334b4f031db1e01508f9ae3e514837e5741d684736dbfa1dbc425fc87",
                "sha256:e2e4ad18baa80ccdbbf6aa5e1976ece30e634931add35077dd4b840e2026c4f2",
                "sha256:10b474e9968443be60978eb2c8b6bf24ae5646640b578acc353b52b1f829065d",
                "sha256:2ef48c8b030568680ca2b6b7b7a38eb929cc30357bfaf0ad61b4e6265b91ee47"
            ]
        }
    }
]

$ docker run --rm -it php:latest curl -I https://services.mailup.com
HTTP/1.1 403 Forbidden
Content-Length: 1233
Content-Type: text/html
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Thu, 19 Jan 2017 18:50:28 GMT

Digital Ocean linux machine:

root@io:~# uname -a
Linux io 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@io:~# docker --version
Docker version 1.13.0, build 49bf474
root@io:~# docker inspect php:latest
[
    {
        "Id": "sha256:97c69bd40b40e1a1338409408ceb61b779097cc857037305450ad9a1ea249695",
        "RepoTags": [
            "php:latest"
        ],
        "RepoDigests": [
            "php@sha256:27c3c78c22adcb782372890fc0a790409ef9a6bdfc30e34b69719e38774c09ec"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2017-01-17T19:11:16.191835022Z",
        "Container": "20a77fd84e12917377c53085f8ba6a5fa1dd7bcac2b9d1407ac52533276bc75a",
        "ContainerConfig": {
            "Hostname": "33842653d6db",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PHPIZE_DEPS=autoconf \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c",
                "PHP_INI_DIR=/usr/local/etc/php",
                "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_LDFLAGS=-Wl,-O1 -Wl,--hash-style=both -pie",
                "GPG_KEYS=A917B1ECDA84AEC2B568FED6F50ABC807BD5DCD0",
                "PHP_VERSION=7.1.0",
                "PHP_URL=https://secure.php.net/get/php-7.1.0.tar.xz/from/this/mirror",
                "PHP_ASC_URL=https://secure.php.net/get/php-7.1.0.tar.xz.asc/from/this/mirror",
                "PHP_SHA256=a810b3f29c21407c24caa88f50649320d20ba6892ae1923132598b8a0ca145b6",
                "PHP_MD5=cf36039303c47f493100afea522a8f53"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"php\" \"-a\"]"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:2826284d23a26e9078f99960f834ebe17561a0b8a152d0398446f3d2ef24b3b7",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "docker-php-entrypoint"
            ],
            "OnBuild": [],
            "Labels": {}
        },
        "DockerVersion": "1.12.3",
        "Author": "",
        "Config": {
            "Hostname": "33842653d6db",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PHPIZE_DEPS=autoconf \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c",
                "PHP_INI_DIR=/usr/local/etc/php",
                "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2",
                "PHP_LDFLAGS=-Wl,-O1 -Wl,--hash-style=both -pie",
                "GPG_KEYS=A917B1ECDA84AEC2B568FED6F50ABC807BD5DCD0",
                "PHP_VERSION=7.1.0",
                "PHP_URL=https://secure.php.net/get/php-7.1.0.tar.xz/from/this/mirror",
                "PHP_ASC_URL=https://secure.php.net/get/php-7.1.0.tar.xz.asc/from/this/mirror",
                "PHP_SHA256=a810b3f29c21407c24caa88f50649320d20ba6892ae1923132598b8a0ca145b6",
                "PHP_MD5=cf36039303c47f493100afea522a8f53"
            ],
            "Cmd": [
                "php",
                "-a"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:2826284d23a26e9078f99960f834ebe17561a0b8a152d0398446f3d2ef24b3b7",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "docker-php-entrypoint"
            ],
            "OnBuild": [],
            "Labels": {}
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 364487897,
        "VirtualSize": 364487897,
        "GraphDriver": {
            "Name": "devicemapper",
            "Data": {
                "DeviceId": "11",
                "DeviceName": "docker-253:0-402775-a4cbf734e15dbae1ee87a56eb253455844dbd4394efd65f973f793b2cd053b58",
                "DeviceSize": "10737418240"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c",
                "sha256:f134b0bf5c584dfb300c39e73b61ea5546b225e6a689e0cb44a45185a25fb95e",
                "sha256:8173a4e80171357ef8a763cbc5a2e24353711fede10d72ca98d867a9a17f6f9f",
                "sha256:b484971334b4f031db1e01508f9ae3e514837e5741d684736dbfa1dbc425fc87",
                "sha256:e2e4ad18baa80ccdbbf6aa5e1976ece30e634931add35077dd4b840e2026c4f2",
                "sha256:10b474e9968443be60978eb2c8b6bf24ae5646640b578acc353b52b1f829065d",
                "sha256:2ef48c8b030568680ca2b6b7b7a38eb929cc30357bfaf0ad61b4e6265b91ee47"
            ]
        }
    }
]
root@io:~# docker run --rm -it php:latest curl -I https://services.mailup.com
HTTP/1.1 403 Forbidden
Content-Length: 1233
Content-Type: text/html
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Thu, 19 Jan 2017 18:52:07 GMT

It also works fine on every AWS and GCP machine I've tried, and yet this fails on every OSX machine I have access to. So I think it's a bit premature to "confirm" that the issue isn't related to Docker for Mac.

dsheets commented 7 years ago

I see the issue in Ubuntu 15.10 with Linux 4.2.0 running Docker 1.10.3 on VMWare Fusion 8.5.3 without Docker for Mac running.

I see the issue on OS X 10.11.6 with homebrew curl installed with brew install curl --force --with-openssl without Docker for Mac running.

This evidence points to Docker for Mac not being involved in your issue.

ijc commented 7 years ago

I ran some tcpdumps on native Linux (succeeds), Docker 4 Mac (fails), osx /usr/bin/curl (uses "SecureTransport", succeeds) and osx homebrew curl with openssl (fails).

The interesting thing I note is that in both the failure cases the initial SSL Client Hello is a 517 byte message which has been TCP segmented into 2 segments (one 512 bytes, one of 5 bytes).

In the Linux success case a similar 517 byte Client Hello message was sent unsegmented.

In the OSX success case (native curl) the Client Hello is only 237 bytes (it contains fewer SSL extension options, only 77 bytes instead of 349 in the other cases) and is unsegmented.

if @dsheets were to do a packet capture on his VMWare fusion case I predict he would see the segmented Client Hello.

So I think this is something to do with the OSX network stack and specifically how it does PMTU/MSS discovery (apparently it starts quite small, leading to segmentation on early TCP messages).

The MTU on all of the my physical network devices on OSX and Linux is 1500.

Googling around a bit I found https://rolande.wordpress.com/2010/12/30/performance-tuning-the-network-stack-on-mac-osx-10-6/ which lead me to sysctl and the net.inet.tcp.mssdflt option which on my OSX system was 512 (the default I presume).

I ran sudo sysctl net.inet.tcp.mssdflt=1024 at which point both of my failing cases (docker and curl w/ openssl) started working. Capturing the traffic I see that the Client Hello is no longer segmented.

So I think this issue is down to a combination of a low default mss on OSX and the fact that services.mailup.com apparently rejects Client Hello messages which are split across multiple TCP segments. I have no idea if this latter behaviour is spec compliant and/or mandated or what.

So I think you could perhaps solve your osx issue by adjusting the mssdflt setting. However you should read up on the full ramifications of doing so and make your own decision because I am not familiar enough with the OSX network stack to say how safe doing so would be. I've set my own system back to 512.

justincormack commented 7 years ago

Are you sure there is not some sort of SSL interception box on the path between you and mailup that can't cope with fragmentation? That is not compliant with the standard, but is quite possible. We are not going to change the OSX default network stack options, we try to run everything in Docker for Mac as non root, and we do not want to interfere with people's global settings.

victortrac commented 7 years ago

@ijc25 Thanks for doing the tcpdump analysis. That was going to be my next step to troubleshoot the issue.

@dsheets @justincormack As shown by @ijc25 , this isn't strictly a "Docker for Mac issue", but it's clearly an issue where identical images behave differently on OSX and Linux. As everyone on OSX will be affected (if not this, then possibly by other subtle network differences), I would argue that this is effectively a Docker for Mac issue, especially since the vast majority of docker users on OSX are simply developing on OSX but deploying on linux and expecting the same behavior.

docker-robott commented 4 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked

docker / for-mac