Cannot connect to Camunda 8 Self-Managed with wildcard SSL certificate

barmac commented 1 year ago

Describe the bug

Given I configure Zeebe to use a wildcard certificate (e.g. *.test.localhost), when I try to deploy a diagram with --zeebe-ssl-certificate=<path>, connection fails.

Zbctl works fine:

❯ zbctl status --address test.test.localhost --certPath=./cert.pem
Cluster size: 1
Partitions count: 1
Replication factor: 1
Gateway version: 8.0.4
Brokers:
  Broker 0 - 0.0.0.0:26501
    Version: 8.0.4
    Partition 1 : Leader, Healthy

Grpc logs:

D 2022-11-30T16:13:12.619Z | subchannel | (2) 127.0.0.1:26500 creating HTTP/2 session
D 2022-11-30T16:13:12.633Z | subchannel | (2) 127.0.0.1:26500 connection closed with error unable to verify the first certificate
D 2022-11-30T16:13:12.633Z | subchannel | (2) 127.0.0.1:26500 connection closed

Perhaps my issue is caused by https://github.com/nodejs/node/issues/14613.

Steps to reproduce

Generate a rootCA + wildcard certificate:
script

#! /bin/bash

# Create root CA & Private key

openssl req -x509 \
            -sha256 -days 356 \
            -nodes \
            -newkey rsa:4096 \
            -subj "/CN=test.localhost/C=US/L=San Fransisco" \
            -keyout rootCA.key -out rootCA.crt

# Generate Private key

openssl genrsa -out cert.key 4096
openssl pkcs8 -topk8 -inform pem -in cert.key -outform pem -nocrypt -out key.pem

# Create csf conf

cat > csr.conf <<EOF
[ req ]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C = US
ST = California
L = San Fransisco
O = MLopsHub
OU = MlopsHub Dev
CN = localhost

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = *.test.localhost

EOF

# create CSR request using private key

openssl req -new -key key.pem -out server.csr -config csr.conf

# Create a external config file for the certificate

cat > cert.conf <<EOF

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = *.test.localhost

EOF

# Create SSl with self signed CA

openssl x509 -req \
    -in server.csr \
    -CA rootCA.crt -CAkey rootCA.key \
    -CAcreateserial -out cert.pem \
    -days 365 \
    -sha256 -extfile cert.conf

Run Zeebe with that cert (e.g. docker run --rm -p 26500:26500 -e ZEEBE_BROKER_NETWORK_HOST=0.0.0.0 -e ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED=true -e ZEEBE_BROKER_GATEWAY_SECURITY_CERTIFICATECHAINPATH=/usr/local/zeebe/cert.pem -e ZEEBE_BROKER_GATEWAY_SECURITY_PRIVATEKEYPATH=/usr/local/zeebe/key.pem --mount type=bind,source="$(pwd)"/cert.pem,target=/usr/local/zeebe/cert.pem --mount type=bind,source="$(pwd)"/key.pem,target=/usr/local/zeebe/key.pem camunda/zeebe:8.0.4)
Configure /etc/hosts to point your docker ip to test.test.localhost
Run Modeler with the flag --zeebe-ssl-certificate=./cert.pem
Try to deploy

Expected behavior

It should work.

Environment

OS: [e.g. MacOS 10.2, Windows 10]
Camunda Modeler Version: [e.g. 2.0.0]
Execution Platform: [e.g. Camunda Platform, Camunda Cloud]
Installed plug-ins: [...]

Additional context

Related to SUPPORT-15142 Related to SUPPORT-15699 Related to SUPPORT-13807

CatalinaMoisuc commented 1 year ago

@barmac could open an issue in zeebe-node and move this in backlog on our side.

barmac commented 1 year ago

I just had an idea how we could potentially fix this in the Modeler but I need to try it out yet.

barmac commented 1 year ago

I can deploy to Zeebe when I use the rootCA.crt in the flag instead of the wildcard certificate signed with the former one: camunda-modeler --zeebe-ssl-certificate='./rootCA.crt'. However, I cannot still just use the cert.pem as I can with zbctl.

barmac commented 1 year ago

I created an issue in zeeb-node: https://github.com/camunda-community-hub/zeebe-client-node-js/issues/288

nikku commented 1 year ago

@barmac regarding the steps to reproduce did you also try adding a commonName wildcard in addition to the subject alt names?

commonName = *.test.localhost

I guess that is deprecated, but could be a workaround.

barmac commented 1 year ago

Not sure, but still we want the Modeler experience to be on par with zbctl.

nikku commented 1 year ago

Examining relevant issues in the nodejs issue tracker I found this comment:

Thanks, I'll close the issue. Per spec, wildcard certificates only match one subdomain. IOW, working as intended.

https://github.com/nodejs/node/issues/13884#issuecomment-310827777

We may want to verify if that is correct:

SUB_ALT_NAME = *.foo.bar -> DOES NOT MATCH woop.abc.foo.bar

barmac commented 1 year ago

Thanks for looking for a solution @nikku. Note that the test case certificates already meet the requirement of a single subdomain, i.e. DNS.1 = *.test.localhost matches CN=test.localhost.

nikku commented 1 year ago

I understood https://github.com/nodejs/node/issues/13884#issuecomment-310827777 differently:

If I define a wildcard certificate for *.foo.bar then it does not match walt.woop.foo.bar but only woop.foo.bar.

This is confirmed via this post.

nikku commented 1 year ago

~~Likely a plain duplicate of https://github.com/camunda/camunda-modeler/issues/3366 with the same root cause.~~

Maybe a duplicate, but different root cause.

nikku commented 1 year ago

I dove into the topic using our zeebe-connection-test. and found the following:

Given a CA certificate root.crt
I obtain a server secret server.key and certificate server.crt signed by root.crt
I spin up zeebe using a server certificate and private key
I connect with zbctl and configure server.crt via certPath. The connection succeeds.
I connect with zeebe-node / Camunda Modeler configuring server.crt. The connection fails.
I connect with zeebe-node / Camunda Modeler configuring root.crt. The connection succeeds.

TLDR: In Camunda Modeler / zeebe-node you must configure a trusted certificate authority, not the actual server certificate.

The Camunda Modeler behavior can be reproduced with openssl, too.

nikku commented 1 year ago

It is still not clear to me who is "in error" here:

zbctl also refers to the certificate that should be passed as ZEEBE_CA_CERTIFICATE_PATH, however the command line argument name is not clear in that regard. The help leaves many more options open:

zbctl --help
[...]
    --certPath string           Specify a path to a certificate with which to validate
                                gateway requests. If omitted, will read from the
                                environment variable 'ZEEBE_CA_CERTIFICATE_PATH'

Camunda Modeler argument is called --zeebe-ssl-certificate, not indicating that it must be the signing root certificate. In the documentation we don't clearly present that, too.

nikku commented 1 year ago

For what it is worth I've used the zebee-connection-test to validate the zbctl behavior (validating the server certificate with the server certificate itself) with openssl and failed:

# server configured with server.crt
$ openssl s_client -CAfile cert/server.crt -connect zeebe.mycorp.com:26500

CONNECTED(00000003)                                                                        
depth=0 C = US, ST = California, L = San Fransisco, O = MLopsHub, OU = MlopsHub Dev, CN = localhost                                                                                   
verify error:num=20:unable to get local issuer certificate                                 
verify return:1                                                                                                                                                                       
depth=0 C = US, ST = California, L = San Fransisco, O = MLopsHub, OU = MlopsHub Dev, CN = localhost                                                                                   
verify error:num=21:unable to verify the first certificate                                 
verify return:1                                                                                                                                                                       
depth=0 C = US, ST = California, L = San Fransisco, O = MLopsHub, OU = MlopsHub Dev, CN = localhost                                                                                   
verify return:1                                                                            
---                                                                                        
Certificate chain                                                                          
 0 s:C = US, ST = California, L = San Fransisco, O = MLopsHub, OU = MlopsHub Dev, CN = localhost                                                                                      
   i:CN = mycorp.com, C = US, L = San Fransisco                                            
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256                                   
   v:NotBefore: Jan 26 22:34:13 2023 GMT; NotAfter: Jan 26 22:34:13 2024 GMT
[...]

To get it work I must explicitly pass the signing authority (root.crt). In that case I can just omit server.crt, too, though:

$ openssl s_client -CAfile cert/server.crt -CAfile cert/root.crt -connect zeebe.mycorp.com:26500                                                                              
CONNECTED(00000003)                                                                        
depth=1 CN = mycorp.com, C = US, L = San Fransisco                                         
verify return:1                                                                                                                                                                       
depth=0 C = US, ST = California, L = San Fransisco, O = MLopsHub, OU = MlopsHub Dev, CN = localhost
verify return:1                                                                            
---                                                                                        
Certificate chain                                                                          
 0 s:C = US, ST = California, L = San Fransisco, O = MLopsHub, OU = MlopsHub Dev, CN = localhost
   i:CN = mycorp.com, C = US, L = San Fransisco                                            
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256                                   
   v:NotBefore: Jan 26 22:34:13 2023 GMT; NotAfter: Jan 26 22:34:13 2024 GMT 
[...]

nikku commented 1 year ago

Raised the server vs. root certificate behavior with the zeebe team.

barmac commented 1 year ago

Hmm so if we are to still use zeebe-node, we should be clear about this limitation, and change both the flag name and the documentation to reflect this.

Still, I don't get it why it should be impossible to accept server's certificate based solely on user's trust in that certificate, not the signing authority.

barmac commented 1 year ago

Still, I don't get it why it should be impossible to accept server's certificate based solely on user's trust in that certificate, not the signing authority.

On the other hand, based on openssl behaviour, it could indeed be disallowed.

nikku commented 1 year ago

Still, I don't get it why it should be impossible to accept server's certificate based solely on user's trust in that certificate, not the signing authority.

Here is why:

The server certificate is signed by a root CA
Camunda Modeler / zeebe-node cannot validate the root CA
Hence the server certificate is invalid

If you don't sign the server certificate via a root CA then it works in Camunda Modeler / zeebe-node / openssl, too.

nikku commented 1 year ago

Proposal to improve our existing flags documentation: https://github.com/camunda/camunda-platform-docs/pull/1652.

nikku commented 1 year ago

@barmac hopped onto the improvement train and contributed eager certificate validation to the modeler: https://github.com/camunda/camunda-modeler/pull/3411.

We now print a meaningful errors when certificates presented are not valid.

nikku commented 1 year ago

Closing this as won't fix.

In the Camunda Modeler certificates used to validate a secured connection to Zeebe must be signing authority root certificates, not the server certificate itself.

We are now eagerly validating the correctness of certificates (https://github.com/camunda/camunda-modeler/pull/3411) and improved our documentation on this matter (https://github.com/camunda/camunda-platform-docs/pull/1652), too.

camunda / camunda-modeler