search

lensesio / fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, , 20+ connectors
https://lenses.io
Apache License 2.0
2.02k stars 333 forks source link

Issue with SSL when using ENABLE_SSL in tandem with ADV_HOST #72

Open telbot opened 6 years ago

telbot commented 6 years ago

I'm having an issue when I attempt to enable SSL using ENABLE_SSL in tandem with ADV_HOST.

If I don't enable SSL, or if I exclude the ADV_HOST in the following, it works without issue.

docker run --rm --net=host \
           -e ENABLE_SSL=1 \
           -e ADV_HOST=192.168.1.81 \
           landoop/fast-data-dev

If I run the above docker container and then execute the following commands as per the readme:

docker run --rm -it --net=host --entrypoint bash landoop/fast-data-dev root@fast-data-dev / $ wget localhost:3030/certs/truststore.jks root@fast-data-dev / $ wget localhost:3030/certs/client.jks root@fast-data-dev / $ kafka-producer-perf-test --topic tls_test \ --throughput 100000 --record-size 1000 --num-records 2000 \ --producer-props bootstrap.servers="localhost:9093" security.protocol=SSL \ ssl.keystore.location=client.jks ssl.keystore.password=fastdata \ ssl.key.password=fastdata ssl.truststore.location=truststore.jks \ ssl.truststore.password=fastdata

I see the following output (loops):

[2018-06-20 13:04:35,094] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 9 : {tls_test=UNKNOWN_SERVER_ERROR} (org.apache.kafka.clients.NetworkClient)
[2018-06-20 13:04:35,202] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 10 : {tls_test=UNKNOWN_SERVER_ERROR} (org.apache.kafka.clients.NetworkClient)
[2018-06-20 13:04:35,309] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 11 : {tls_test=UNKNOWN_SERVER_ERROR} (org.apache.kafka.clients.NetworkClient)
[2018-06-20 13:04:35,418] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 12 : {tls_test=UNKNOWN_SERVER_ERROR} (org.apache.kafka.clients.NetworkClient)

When I use the following, excluding ADV_HOST, it seems to work fine:

docker run --rm --net=host \
           -e ENABLE_SSL=1 \
           landoop/fast-data-dev

This is the output I see:

someone@ubuntu:~/scripts$ sudo docker run --rm -it --net=host --entrypoint bash landoop/fast-data-dev
root@fast-data-dev / $ wget localhost:3030/certs/truststore.jks
--2018-06-20 13:08:52--  http://localhost:3030/certs/truststore.jks
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:3030... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1086 (1.1K) [application/octet-stream]
Saving to: 'truststore.jks'

     0K                                    100%  224M=0s

2018-06-20 13:08:52 (224 MB/s) - 'truststore.jks' saved [1086/1086]

root@fast-data-dev / $ wget localhost:3030/certs/client.jks
--2018-06-20 13:09:03--  http://localhost:3030/certs/client.jks
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:3030... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2328 (2.3K) [application/octet-stream]
Saving to: 'client.jks'

     0K                                    100%  541M=0s

2018-06-20 13:09:03 (541 MB/s) - 'client.jks' saved [2328/2328]

root@fast-data-dev / $ kafka-producer-perf-test --topic tls_test \
>   --throughput 100000 --record-size 1000 --num-records 2000 \
>   --producer-props bootstrap.servers="localhost:9093" security.protocol=SSL \
>   ssl.keystore.location=client.jks ssl.keystore.password=fastdata \
>   ssl.key.password=fastdata ssl.truststore.location=truststore.jks \
>   ssl.truststore.password=fastdata
[2018-06-20 13:09:12,844] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {tls_test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
2000 records sent, 2453.987730 records/sec (2.34 MB/sec), 120.25 ms avg latency, 577.00 ms max latency, 113 ms 50th, 189 ms 95th, 203 ms 99th, 208 ms 99.9th.

Note that despite the errors with ADV_HOST property, I can create topics fine - it seems that when I listen to or attempt t consume I see the errors:

root@fast-data-dev / $ kafka-topics.sh --create \
>   --zookeeper localhost:2181 \
>   --replication-factor 1 --partitions 13 \
>   --topic my-topic
bash: kafka-topics.sh: command not found
root@fast-data-dev / $ kafka-topics --create \
>   --zookeeper localhost:2181 \
>   --replication-factor 1 --partitions 1 \
>   --topic davids-topic
Created topic "davids-topic".
root@fast-data-dev / $ kafka-console-producer \
>     --broker-list 192.168.1.81:9092 \
>     --topic spring-topic
>jxjjxjxjxj
[2018-06-20 13:16:47,916] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {spring-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>Trying again...

Any ideas?