[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate when using docker compose with DOCKER_HOST

ivictbor commented 4 years ago

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Output of docker version

Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Fri Feb 28 23:45:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Wed Feb 19 01:06:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

Output of docker-compose config (Make sure to add the relevant -f and other flags)

does not matter

Steps to reproduce the issue

I want to use docker-compose with DOCKER_HOST setting and custom self signed certificates to deploy to another host
I copied ca.pem cert.pem key.pem generated on my server to ~/.docker
did export DOCKER_HOST=tcp://my.host:2376 DOCKER_TLS_VERIFY=1

did docker ps and it works perfectly:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Observed result

Then, executing

docker-compose ps
ERROR: SSL error: HTTPSConnectionPool(host='my.host', port=2376): Max retries exceeded with url: /v1.30/volumes/deploy_v-db-data (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1076)')))

Expected result

docker-compose ps works without an error and using same Environment variables

Additional information

Checked another env variables which might prevent requests used in dockercompose to work wrong like echo $REQUESTS_CA_BUNDLE, they are empty.

How to debug it? Thanks

ckotte commented 3 years ago

Does it work if you execute CURL_CA_BUNDLE='' docker-compose ps?

ivictbor commented 3 years ago

Does it work if you execute CURL_CA_BUNDLE='' docker-compose ps?

Thanks a lot for reply, seams like this env var was also empty, but I will recheck soon and let know.

For now we use ssh url in docker_host which seams to be slower.

@ckotte Most interesting fact for me - why does it work perfectly when certs are generated by docker-machine? I tried and all environment variables were same! Only differece is that I generated certificates with openssl.

ckotte commented 3 years ago

Maybe it's an openssl issue? I could fix it by downgrading openssl from 1.1.1h to 1.1.1g

# docker-compose version
docker-compose version 1.27.4, build unknown
docker-py version: 4.3.1
CPython version: 3.8.5
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

jankatins commented 3 years ago

I had the same error (running against qnap nas, which gave me the cert bundle to install in ~/.docker) and the following fixed it for me. I run a debian unstable on my laptop and just updated docker-compose with pip.

Does it work if you execute CURL_CA_BUNDLE='' docker-compose ps?

ckotte commented 3 years ago

@jankatins What versions are you using? Downgrading docker-compose didn't work on my side - only downgrading openssl... I also use a QNAP NAS. I suspect the certs are invalid and a change introduced in openssl 1.1.1h could be the issue

jankatins commented 3 years ago

QNAP is on latest version for my NAS (4.4.3.1444) and docker compose is 1.27.4. OS on laptop is debian unstable. Openssl debian package is openssl 1.1.1h-1:

$ docker-compose version
docker-compose version 1.27.4, build unknown
docker-py version: 4.3.1
CPython version: 3.8.6
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

ckotte commented 3 years ago

I bet it works if you downgrade openssl. I think it's related to this change: "Disallow explicit curve parameters in verifications chains when X509_V_FLAG_X509_STRICT is used". The docker certificates on QNAP are regenerated automatically before they expire, but I couldn't find a way to regenerate them manually. It's probably an issue with the certificates and maybe it could be resolved with new certificates...

krugerm-4c commented 3 years ago

Hi, I am also having this issue where I followed the official Docker Documentation to setup TLS on the Docker Daemon (https://docs.docker.com/engine/security/https/), but I'm also getting the [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate issue.

Surely there needs to be a way to point docker-compose to the generated certificates to allow it?

I honestly don't think setting CURL_CA_BUNDLE='' is a viable option as this technically circumvents the verify check.

ckotte commented 3 years ago

@krugerm-4c I didn't say CURL_CA_BUNDLE="" is a valid option. It's just a workaround to be able to use docker-compose. What's your workaround??? May I ask what openssl version you are using?

krugerm-4c commented 3 years ago

@ckotte For the time being I am using the CURL_CA_BUNDLE environment variable just to be able to continue with my work. Currently I have the following OpenSSL versions (docker-compose and host-level respectively).

docker-compose version 1.27.4, build 40524192 docker-py version: 4.3.1 CPython version: 3.7.7 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019

OpenSSL 1.0.2k-fips 26 Jan 2017

Unfortunately I am not in a position to change the host's openssl version as this is approved and managed by our security team. Thus I am looking for an elegant solution from the application's (docker-compose) perspective without having to compromise security elements.

ivictbor commented 3 years ago

@krugerm-4c Just in case, might be useful for you - when I faced the issue, I had really no time to properly investigate it, so now I am using plain SSH in DOCKER_HOST, details are here if you need it: https://hinty.io/ivictbor/deploy-docker-compose-using-drone-ci/ We are using it for tens prod instances and I really satisfied, so even don't want to spend time on certificates now.

Pros:

Really simple setup, what you need is just export DOCKER_HOST=ssh://root@ip and no certificates staff at all, no docker machines, plus couple of lines in ssh config to accept multiple connections (added in post above).
SSH is pretty secure way to connect

Drawbacks:

Might be slower, takes several additional second for each layer to build? But not a big deal if you set up layers in Dockerfiles in a right way.

hholst80 commented 3 years ago

I can reproduce this with just docker-py version: 4.3.1

import docker
print("docker-py version: %s" % docker.__version__)
client = docker.from_env()

sceptic30 commented 3 years ago

Most likely this is a bug that needs to be corrected. My best guess is that docker-compose is not passing (or passing correctly) the DOCKER_CERT_PATH variable, from the Docker environment variables. Anyone managed to get docker-compose to work?

krugerm-4c commented 3 years ago

Most likely this is a bug that needs to be corrected. My best guess is that docker-compose is not passing (or passing correctly) the DOCKER_CERT_PATH variable, from the Docker environment variables. Anyone managed to get docker-compose to work?

Nope. This has still been an issue for me.

hholst80 commented 3 years ago

docker compose uses docker-py, and docker-py has the issue. its a problem upstream from docker-compose.

the workaround is to set export DOCKER_CERT_PATH="$HOME/.docker"

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically closed because it had not recent activity during the stale period.

Norman416 commented 2 years ago

Please check ca-cert.pem and server.pem.

thohng commented 2 years ago

Confirm that there is no issues on certificates with docker and docker-compose.

This is my default setup which put client certs at folder %USERPROFILE%\.docker

Docker compose v1: docker-compose version 1.29.2, build 5becea4c

We need to make sure correct self-signed certificates both client and server, such as: client.pem, ca.pem, server.pem,etc. Better should follow the document https://docs.docker.com/engine/security/protect-access/ https://docs.docker.com/engine/security/certificates/

ivictbor commented 2 years ago

BTW we fixed our issue and even created a script which generates TLS certificates , without docker machine.

Basically you can execute it on server:

curl -s -L https://raw.githubusercontent.com/devforth/docker-tls-generator/main/generate-tls.sh | bash

And then deliver content of the machine from which you want to connect to Docker

cat ~/.docker/ca.pem
cat ~/.docker/key.pem
cat ~/.docker/cert.pem

💡 make sure that you run the script on each new server because it uses public IP address of server as Host in certificate (script does curl ifconfig.me)

💡 Tested on Ubuntu 20.04, works out of the box

We are using it to deploy from our CI server, here is detailed guide:

https://hinty.io/vverenko/deploy-docker-compose-using-woodpecker-ci/

So I think the issue is resolved and could be closed

docker / compose