TLS Issues setting up own Notary Server

[JensVD]

Hello

We are trying to set up our own notary server but keep running into the error 'transport: x509: certificate signed by unknown authority'.

We have been following the guidelines that were suggested and generated our own certificate (chain). We now have our own 'root-ca.crt', 'intermediate-ca.crt', 'notary-server.crt' and 'notary-signer.crt'.

The configuration for the notary-server is like this:

{
        "server": {
                "http_addr": ":4443",
                "tls_key_file": "./notary-server.key",
                "tls_cert_file": "./notary-server.crt"
        },
        "trust_service": {
                "type": "remote",
                "hostname": "notary-signer",
                "port": "7899",
                "tls_ca_file": "./root-ca.crt",
                "key_algorithm": "rsa",
                "tls_client_cert": "./notary-server.crt",
                "tls_client_key": "./notary-server.key"
        },
        "logging": {
                "level": "debug"
        },
        "storage": {
                "backend": "mysql",
                "db_url": "server@tcp(mysql:3306)/notaryserver?parseTime=True"
        }
}

And the signer config:

{
        "server": {
                "grpc_addr": ":7899",
                "tls_cert_file": "./notary-signer.crt",
                "tls_key_file": "./notary-signer.key",
                "client_ca_file": "./notary-server.crt"
        },
        "logging": {
                "level": "debug"
        },
        "storage": {
                "backend": "mysql",
                "db_url": "signer@tcp(mysql:3306)/notarysigner?parseTime=True"
        }
}

To be honest, we just tried to copy what the development version did but with our own certificates, which means that the certificate chain is as follows: The root-ca.crt is selfsigned, the intermediate-ca.crt is signed by the root and the notary-server.crt & notary-signer.crt is signed by the intermediate-ca.crt.

Now when starting the services the following entry keeps appearing in the notary-server logs:

{"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority\"","time":"2019-05-21T05:15:49Z"}

And in the notary-signer this appears one on startup:

2019/05/21 05:13:09 grpc: Server.Serve failed to complete security handshake from "172.19.0.3:36800": remote error: tls: bad certificate

Any help would be appreciated as I've spent a lot of time on this but still haven't figured out what I've been doing wrong. If any more information is required, please just ask.

Thank you!

[Jmarqua1]

I've been struggling with this for some time now in a local test set up. Right now, I've reverted back to a fresh clone with their original CAs but I still had these same errors even with a fresh clone! And later, I would have more issues with the docker client when trying push. It was due to a company proxy and proxy settings.

So for people debugging this, if you have a proxy, make sure all the Dockerfiles in the project have the ENV https://myproxy set. If yours requires https/http to be set, make sure its in there. It may just need myproxy with out the https://. That's all dependent on your proxy and how it resolves.

Additionally, I've found when using the notary CLI as well as docker CLI, it doesn't always pick up on no_proxy/NO_PROXY environment variables or even if you specify it in docker's proxy config files. If you are running things internally for both your registry and notary server, try dropping into a new shell without any proxy environment variables set.

Lastly, some things I've tried for verifying my certs from the notary server or signer containers:

openssl s_client -connect <signer or server>:port -CAfile <myCA.crt> -verify_hostname <hostname of signer or server>

You can use this to verify your cert and the host name. With out the -veryfiy_hostname flag, openssl does not check that the hostname you connected to matches what the cert is for. You can also use curl and pass in the CA.crt file for additional testing.

Just know that a bad cert error might not be a cert error after all, it could be your proxy. Good luck.

[dkulchinsky]

Hey @JensVD!

I realize it's been a while since you opened this issue, but I'm hitting a similar problem and can't seem to find a solution.

there's a bunch of details here on my specific issue: https://github.com/bitnami/charts/issues/3545

We're using self-signed CA and Server certificates created by the Helm chart during deployment and getting the same error on the notary server when trying to connect to the notary signer, I have verified the CA and notary-signer Certs in every possible way and it all checks out, also connecting with curl or openssl s_client from the notary-server pod to the notary-signer service works just fine.

pretty much losing hope here... if you had a chance to sort this out it'd be great if you could chime in with some details.

@Jmarqua1 thanks for sharing this as well! I think it's worth mentioning that, if I understand correctly, your solution works for the docker/notary client communication with the notary-server, while what @JensVD was reporting (and the issue I'm stuck with now as well) is the internal communication issue between the notary-server and notary-signer (which uses a separate set of certificates).

[bj1074]

Anyupdate on this issue ?? similar issue I'm also facing

[bj1074]

@dkulchinsky @JensVD

[JensVD]

@bj1074 No, we gave up our notary project and now use harbor as our Docker registry which includes the notary service

[IAL32]

@bj1074 Anyupdate on this issue ?? similar issue I'm also facing

These errors can happen for a couple of reasons:

1. The certificate itself has expired (not the case for this particular issue) and needs to be renewed/reissued
2. The Notary Signer is served on an IP or Domain Name that is not included in the certificate.

To solve 2 you need to either reissue the certificate with the IP or Domain Name to the one where you want to serve the Notary Signer. Please consider looking at fixtures/regenerateTestingCerts.sh, in particular regenerateTestingCerts:67 and regenerateTestingCerts:90 and adjust your signing process accordingly, or serve the Notary Signer from a location it is already signed. In the regenerateTestingCerts.sh file the allowed domain names are:

• Notary Server: subjectAltName = DNS:notary-signer, DNS:notarysigner, DNS:localhost, IP:127.0.0.1
• Notary Signer: subjectAltName = DNS:notary-server, DNS:notaryserver, DNS:localhost, IP:127.0.0.1

In this particular issue, I see that OP is trying to connect to 172.19.0.3:36800, which is not included in the list of allowed Domain Names or IPs, which leads to the specific error that he is having.