Open dariodariodario opened 4 years ago
This looks like an error with how you're passing in the certificate stuff. In particular, your chain may be setup incorrect, which is completely out of control of this library. If you can provide more details that proves the problem is with this library and not your config, then happy to look at this further, but until then I'm 95% sure this is a user/configuration error.
Hi! @jeffwidman I faced with the same issue. We are using strimzi kafka operator for k8s which creates self-signed certificates to authenticate users over tls listener. Originally issue was caused by value of _SSLContext.verifymode of ssl lib. Kafka python conn.py provides 2 options: CERT_OPTIONAL or CERT_REQUIRED in both cases based on SSL lib documentation :
any verification error immediately aborts TLS handshake
In case of self-signed cert we can observe this validation error with openssl tool:
openssl s_client -connect my-cluster-kafka-bootstrap:9093
..... ( output suppressed)
---
SSL handshake has read 5054 bytes and written 452 bytes
Verification error: self signed certificate in certificate chain
.....
Verify return code: 19 (self signed certificate in certificate chain)
The same error returns python script during execution.
My proposal is to add one more parameter let's say _skip_tls_verifications_ to enable
self._ssl_context.verify_mode = ssl.CERT_NONE
for developers purposes ONLY.
Thanks!
I'm getting this same error. Python 3.7.6. kafka-python==2.0.1. BUT I am running this with a local deployment of kafka. So user configuration as @jeffwidman suggested may not be the root cause. Any help is appreciated!
producer = KafkaProducer(bootstrap_servers=['localhost:2181'],
security_protocol="SSL",
ssl_check_hostname=False)
Traceback (most recent call last):
File "performance.py", line 24, in <module>
ssl_check_hostname=False)
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/producer/kafka.py", line 382, in __init__
**self.config)
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/client_async.py", line 242, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/client_async.py", line 907, in check_version
version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 1228, in check_version
if not self.connect_blocking(timeout_at - time.time()):
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 337, in connect_blocking
self.connect()
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 426, in connect
if self._try_handshake():
File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 505, in _try_handshake
self._sock.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 0] Error
Try this: python -m pip install azure-eventhub-checkpointstoreblob-aio --trusted-host=pypi.python.org --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
@henriavo this is the config that i use for Producer pip freeze:
confluent-kafka==1.5.0
pyOpenSSL==19.1.0
on Python 3.8.6
config = {
"bootstrap_server": "ssl://127.0.0.1:9094"
"security.protocol": "SSL",
"ssl.ca.location": "../ssl/root/intermediate/ca-chain.cert.pem",
"ssl.certificate.location": "../ssl/root/intermediate/producer/producer.cert.pem",
"ssl.key.location": "../ssl/root/intermediate/producer/producer.key.pem",
"ssl.key.password": "qwerty"}
ca-chain is intermediate-certificate + root-certificate, producer.cert is signed with the intermediate certificate. It worked for me. Also, enable the DEBUG logs on the broker to get more info on the errors you're getting. The errors helped me a lot in debugging SSL problems. This is the configuration I use for brokers(on docker):
image: confluentinc/cp-kafka:5.5.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
volumes:
- ./ssl/root/intermediate/broker:/etc/kafka/secrets
ports:
- 9092:9092
- 9093:9093
- 9094:9094
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
KAFKA_ZOOKEEPER_SSL_CLIENT_ENABLE: 'true'
KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET: org.apache.zookeeper.ClientCnxnSocketNetty
KAFKA_ZOOKEEPER_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker.keystore.jks
KAFKA_ZOOKEEPER_SSL_KEYSTORE_PASSWORD: confluent
KAFKA_ZOOKEEPER_SSL_KEYSTORE_TYPE: PKCS12
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/broker.truststore.jks
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_PASSWORD: confluent
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,SSL:SSL,PLAINTEXT:PLAINTEXT,SSL_INTERNAL:SSL
KAFKA_ADVERTISED_LISTENERS: INTERNAL://localhost:9092,SSL://localhost:9094,SSL_INTERNAL://broker:9093
KAFKA_LOG4J_LOGGERS: "kafka.authorizer.logger=INFO"
KAFKA_LOG4J_ROOT_LOGLEVEL: INFO
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_DELETE_TOPIC_ENABLE: 'true'
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_DEFAULT_REPLICATION_FACTOR: 1
KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker.keystore.jks
KAFKA_SSL_KEYSTORE_FILENAME: broker.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: broker_creds
KAFKA_SSL_KEYSTORE_PASSWORD: confluent
KAFKA_SSL_KEY_CREDENTIALS: broker_creds
KAFKA_SSL_KEY_PASSWORD: confluent
KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/broker.truststore.jks
KAFKA_SSL_TRUSTSTORE_FILENAME: broker.truststore.jks
KAFKA_SSL_TRUSTSTORE_PASSWORD: confluent
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker_creds
KAFKA_SSL_CLIENT_AUTH: "required"
# KAFKA_OPTS: "-Djavax.net.debug=all"
KAFKA_OPTS is used to enable debug on the broker side. You can enable just ssl debug with something like -Djavax.net.debug=ssl(if i remember correctly)
Hi @jeffwidman , I am facing the same issue with strimzi. How did you solve yours? Thanks
Hello, I'm facing the same issue with strimzi right now.
I'm currently using the sslcontext
as a workaround
cert = "user.crt"
key = "user.key"
context = ssl.create_default_context()
context.load_cert_chain(certfile=cert, keyfile=key)
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
self.ssl_context = context
self.security_protocol = "SSL"
@ImadYamane @alekskar how did you please solve the issue when using Strimzi self signed certs ? And what Strimzi files did you use as ssl_cafile
and ssl_certfile
?
it is significant where you get your ssl_cafile
from
when i was using ca.crt
( from user's secret ) as ssl_cafile
then I get an error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain
, when I switched to the ca.crt
from secret cluster-ca-cert
it worked like a charm
Thank you @dechoma . If anybody is using Strimzi Kafka Operator then you should take ca.crt from cluster-cluster-ca-crt
and not from cluster-clients-ca-crt
.
for thoes suffer from the problem and use gmail account to download some files or use to generate some confidential files, try to use another email account other than *@gmail.com.
Estou obtendo o seguinte erro:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:992)
Quando tento me conectar ao broker(Mosquitto que está no docker, ou sejá, um Pseudo-Broker) onde está utilizando o SSL. Gerei minha CA, certificados e chaves para o cliente e servidor e e está devidamente configurada no mosquito.conf, e só apresenta erro na conexão usando o PAHO ou MQTTX.
Alguem poderia ajudar?
Hello, I am using kafka-python>=2.0.1 (with Python 3.7.6 on MacOsx, build from the python website) it generally works fine, but when I tried to access our broker, that has SSL auth, I will get the above error. it happens if I enable SSL with my certificates. I'm sure that there's one that is self signed internally in my company.
with this, when sending a message, it gets stuck in a loop and prints:
to solve this I tried a number of python installations (provided by brew, pyenv and eventually the installer from the python website). I tried to fix the issue by running
Install Certificates.command
that reinstalls the certificates. but it keeps doing so.If I remove
ssl_cafile
andssl_certfile
(or just one of the two, leavingssl_keyfile
) it will stop giving that exception, but will just die withCan anyone help please?