search

appsody / stacks

Appsody application stacks. This repo will be archived soon.
https://appsody.dev
Apache License 2.0
89 stars 121 forks source link

Nodejs-express Kafka stack failing to resolve Event Streams certificates #836

Closed djones6 closed 4 years ago

djones6 commented 4 years ago

Describe the bug I recently repackaged an existing Express microservice (using node-rdkafka) on top of the nodejs-express stack.

This works fine with Kafka over a plain connection (no TLS), or with a TLS-enabled Kafka where the certificate is explicitly provided to the client (supplying a .crt via the ssl.ca.location configuration property). However, when trying to connect to a managed Event Streams on IBM Public Cloud, it fails with:

[LibrdKafkaError: Local: SSL error] {
  origin: 'local',
  message: 'ssl error',
  code: -1,
  errno: -1
}

After debugging this quite a lot further, I've narrowed it down to a failure to verify the broker's certificate. I gather that this relies on the TLS SNI extension, so maybe there is a problem in that area with the node:12-slim image that is used to build the production image for this stack.

I can reproduce the failure directly inside the container using the openssl command line tool.

To Reproduce

docker run -it node:12-slim /bin/bash
apt-get update && apt-get install openssl
openssl s_client -connect broker-3-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com -port 9093 -servername broker-3-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com

Expected behavior (you can get this output by running the openssl command above locally)

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = eventstreams.cloud.ibm.com
verify return:1
---
Certificate chain
 0 s:/CN=eventstreams.cloud.ibm.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Actual behaviour

CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/CN=eventstreams.cloud.ibm.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Environment Details (please complete the following information):

If applicable please specify:

djones6 commented 4 years ago

Update - looks like the node:12-slim image is missing the ca-certificates package. Without this, there are no trusted roots available to verify public certificates.

This problem is likely not specific to the node stack, though I haven't checked on any others.

gireeshpunathil commented 4 years ago

do you know if images of other runtimes suffer from the same issue? If not, it may be worthwhile to raise it up with the upstream (https://github.com/nodejs/docker-node/) .

djones6 commented 4 years ago

@gireeshpunathil It's only an issue with the slim image. The documentation for the slim image (https://github.com/nodejs/docker-node/#nodeslim) implies that, by design, only the minimum set of packages required to run node are included.

The question is then whether we should start from this minimal set and require stack creators to add packages that are needed by more sophisticated applications, or to start from a richer set (ie. the default images, not slim) which are larger, but less likely to require a stack modification.

I feel like Appsody's direction is more the former, where stacks should explicitly define the dependencies rather than inheriting a bunch from a base image. That's fine, but something as basic as being able to connect to external services over TLS seems like it should be part of a base Appsody stack, as the intention is to enable the creation of microservices, and connecting to other services would seem a common requirement.

Even that argument could be countered, for example, I might develop a microservice that only talks to a messaging backbone and does not require SSL verification using public CAs. However, the sheer amount of time taken to debug the absence of this package makes me strongly in favour of including it in the stack image.