`amqp` sink TLS doesn't work

[Fiery-Fenix]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

TLS connection to AMQP broker doesn't work even after #16504 fix. Steps to reproduce:

1. Generate certificates using this script:

   #!/bin/bash
   mkdir certificates
   git clone https://github.com/michaelklishin/tls-gen tls-gen
   cd tls-gen/basic
   make PASSWORD=
   make verify
   make info
   cp result/* -R ../../certificates/
   ls -l ../../certificates

2. Create sample RabbitMQ config (rabbitmq.conf)

   ## Defaults
   default_vhost = /
   default_user = test
   default_pass = test
   ## TLS
   listeners.ssl.default = 5671
   ssl_options.cacertfile = /etc/ssl/rabbit/cacert.pem
   ssl_options.certfile = /etc/ssl/rabbit/cert.pem
   ssl_options.keyfile = /etc/ssl/rabbit/key.pem
   ssl_options.verify = verify_peer
   ssl_options.fail_if_no_peer_cert = false
   ### Enable only actual TLS versions
   ssl_options.versions.1 = tlsv1.3
   ssl_options.versions.2 = tlsv1.2
   ### Set ciphers strict order
   ssl_options.honor_cipher_order = true
   ssl_options.honor_ecc_order = true
   ### Set only good ciphers
   ssl_options.ciphers.1 = TLS_AES_256_GCM_SHA384
   ssl_options.ciphers.2 = TLS_AES_128_GCM_SHA256
   ssl_options.ciphers.3 = TLS_CHACHA20_POLY1305_SHA256
   ssl_options.ciphers.4 = TLS_AES_128_CCM_SHA256
   ssl_options.ciphers.5 = TLS_AES_128_CCM_8_SHA256
   ssl_options.ciphers.6 = ECDHE-ECDSA-AES256-GCM-SHA384
   ssl_options.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
   ssl_options.ciphers.8 = ECDH-ECDSA-AES256-GCM-SHA384
   ssl_options.ciphers.9 = ECDH-RSA-AES256-GCM-SHA384
   ssl_options.ciphers.10 = DHE-RSA-AES256-GCM-SHA384
   ssl_options.ciphers.11 = DHE-DSS-AES256-GCM-SHA384
   ssl_options.ciphers.12 = ECDHE-ECDSA-AES128-GCM-SHA256
   ssl_options.ciphers.13 = ECDHE-RSA-AES128-GCM-SHA256
   ssl_options.ciphers.14 = ECDH-ECDSA-AES128-GCM-SHA256
   ssl_options.ciphers.15 = ECDH-RSA-AES128-GCM-SHA256
   ssl_options.ciphers.16 = DHE-RSA-AES128-GCM-SHA256
   ssl_options.ciphers.17 = DHE-DSS-AES128-GCM-SHA256

3. Run RabbitMQ container using docker-compose:

   version: "3"
   services:
   rabbit:
   image: rabbitmq:3.11-management-alpine
   ports:
   - "5671:5671"
   volumes:
     - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
     - ./certificates/ca_certificate.pem:/etc/ssl/rabbit/cacert.pem
     - ./certificates/server_certificate.pem:/etc/ssl/rabbit/cert.pem
     - ./certificates/server_key.pem:/etc/ssl/rabbit/key.pem

4. Convert PEM encoded certificates to DER encoded (see #16888 bug):

   openssl x509 -in ./certificates/ca_certificate.pem -out ./certificates/ca_certificate.crt -outform der
   openssl x509 -in ./certificates/client_$HOSTNAME_certificate.pem -out ./certificates/client_$HOSTNAME_certificate.crt -outform der
   openssl rsa -in ./certificates/client_$HOSTNAME_key.pem -out ./certificates/client_$HOSTNAME_key.crt -outform der

5. Run vector using provided config

Configuration

# Set global options
data_dir = "/var/lib/vector"
timezone = "UTC"

[api]
enabled = true
address = "127.0.0.1:8686"

[sources.internal_metrics]
type = "internal_metrics"
scrape_interval_secs = 1

[transforms.m2l]
type = "metric_to_log"
inputs = ["internal_metrics"]
host_tag = "host"

[sinks.amqp_collector]
type = "amqp"
inputs = ["m2l"]
connection_string = "amqps://test:test@localhost?timeout=10"
exchange = "test"
[sinks.amqp_collector.tls]
ca_file = "certificates/ca_certificate.crt"
#crt_file = "certificates/client_$HOSTNAME_certificate.crt"
#key_file = "certificates/client_$HOSTNAME_key.crt"
verify_certificate = true
verify_hostname = true
[sinks.amqp_collector.encoding]
codec = "json"

Version

vector 0.29.0 (x86_64-unknown-linux-gnu 1de94ee 2023-03-21)

Debug Output

2023-03-21T15:16:38.164919Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-03-21T15:16:38.164982Z  INFO vector::app: Log level is enabled. level="vector=trace,codec=trace,vrl=trace,file_source=trace,tower_limit=trace,rdkafka=trace,buffers=trace,lapin=trace,kube=trace"
2023-03-21T15:16:38.169552Z  INFO vector::app: Loading configs. paths=["vector.toml"]
2023-03-21T15:16:38.170496Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-03-21T15:16:38.171358Z DEBUG vector::topology::builder: Building new source. component=internal_metrics
2023-03-21T15:16:38.171637Z DEBUG vector::topology::builder: Building new transform. component=m2l
2023-03-21T15:16:38.171836Z DEBUG vector::topology::builder: Building new sink. component=amqp_collector
2023-03-21T15:16:38.175008Z ERROR vector::topology: Configuration error. error=Sink "amqp_collector": creating amqp producer failed: stream did not contain valid UTF-8

Example Data

No response

Additional Context

No response

References

16888

[pront]

Added repro here: https://github.com/vectordotdev/issue-examples/tree/main/issues/16890