durandom
commented
2 years ago I was missing these steps: https://github.com/b4mad/formula1-telemetry-kafka/tree/master/deployment/openshift#run-the-f1-telemetry-udp-to-apache-kafka-locally
export KAFKA_BOOTSTRAP_SERVERS=$(oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.type=="external")].bootstrapServers}{"\n"}')
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.p12}' | base64 -d > ca.p12
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 -d > ca.password
KAFKA_BOOTSTRAP_SERVERS=my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443 KAFKA_TRUSTSTORE_LOCATION=$(pwd)/ca.p12 KAFKA_TRUSTSTORE_PASSWORD=$(cat ca.password) java -jar udp-kafka/target/f1-telemetry-udp-kafka-1.0-SNAPSHOT-jar-with-dependencies.jar
2021-11-05 12:19:46 INFO F1UdpKafkaApp:23 - Config: F1UdpKafkaAppConfig[udpPort=20777, kafkaBootstrapServers=my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443, kafkaTruststoreLocation=/Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12, kafkaTruststorePassword=Gsy8gGDwKEpX, f1DriversTopic=f1-telemetry-drivers, f1EventsTopic=f1-telemetry-events, f1RawPacketsTopic=f1-telemetry-packets]
2021-11-05 12:19:46 INFO RawPacketsRouteBuilder:28 - KafkaEndpoint = kafka:f1-telemetry-packets?brokers=my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443&clientId=raw-packets&sslTruststoreLocation=/Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12&sslTruststorePassword=Gsy8gGDwKEpX&sslTruststoreType=PKCS12&securityProtocol=SSL
2021-11-05 12:19:46 INFO EventsRouteBuilder:36 - KafkaEndpoint = kafka:f1-telemetry-events?brokers=my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443&clientId=events&valueSerializer=io.ppatierno.formula1.EventSerializer&sslTruststoreLocation=/Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12&sslTruststorePassword=Gsy8gGDwKEpX&sslTruststoreType=PKCS12&securityProtocol=SSL
2021-11-05 12:19:46 INFO DriversRouteBuilder:42 - KafkaEndpoint = kafka:f1-telemetry-drivers?brokers=my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443&clientId=drivers&valueSerializer=io.ppatierno.formula1.DriverSerializer&sslTruststoreLocation=/Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12&sslTruststorePassword=Gsy8gGDwKEpX&sslTruststoreType=PKCS12&securityProtocol=SSL
2021-11-05 12:19:46 WARN NettyConfiguration:154 - The decoder io.ppatierno.formula1.PacketEventDecoder@27c86f2d is not @Shareable or an ChannelHandlerFactory instance. The decoder cannot safely be used.
2021-11-05 12:19:47 INFO ProducerConfig:361 - ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443]
buffer.memory = 33554432
client.dns.lookup = use_all_dns_ips
client.id = drivers
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
internal.auto.downgrade.txn.commit = false
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 127000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class io.ppatierno.formula1.DriverSerializer
2021-11-05 12:19:47 INFO AppInfoParser:119 - Kafka version: 2.7.0
2021-11-05 12:19:47 INFO AppInfoParser:120 - Kafka commitId: 448719dc99a19793
2021-11-05 12:19:47 INFO AppInfoParser:121 - Kafka startTimeMs: 1636111187355
2021-11-05 12:19:47 INFO NettyComponent:164 - Creating shared NettyConsumerExecutorGroup with 25 threads
2021-11-05 12:19:47 INFO ProducerConfig:361 - ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443]
buffer.memory = 33554432
client.dns.lookup = use_all_dns_ips
client.id = raw-packets
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
internal.auto.downgrade.txn.commit = false
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 127000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
2021-11-05 12:19:47 INFO AppInfoParser:119 - Kafka version: 2.7.0
2021-11-05 12:19:47 INFO AppInfoParser:120 - Kafka commitId: 448719dc99a19793
2021-11-05 12:19:47 INFO AppInfoParser:121 - Kafka startTimeMs: 1636111187453
2021-11-05 12:19:47 INFO ProducerConfig:361 - ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:443]
buffer.memory = 33554432
client.dns.lookup = use_all_dns_ips
client.id = events
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
internal.auto.downgrade.txn.commit = false
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 127000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /Users/mhild/src/github/durandom/formula1-telemetry-kafka/ca.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class io.ppatierno.formula1.EventSerializer
2021-11-05 12:19:47 INFO AppInfoParser:119 - Kafka version: 2.7.0
2021-11-05 12:19:47 INFO AppInfoParser:120 - Kafka commitId: 448719dc99a19793
2021-11-05 12:19:47 INFO AppInfoParser:121 - Kafka startTimeMs: 1636111187520
2021-11-05 12:19:47 INFO AggregateProcessor:1493 - Defaulting to MemoryAggregationRepository
2021-11-05 12:19:47 INFO AggregateProcessor:1561 - Using CompletionTimeout to trigger after 10000 millis of inactivity.
2021-11-05 12:19:47 INFO SingleUDPNettyServerBootstrapFactory:192 - ConnectionlessBootstrap binding to 0.0.0.0:20777
2021-11-05 12:19:47 INFO NettyConsumer:77 - Netty consumer bound to: 0.0.0.0:20777
2021-11-05 12:19:47 INFO AbstractCamelContext:2980 - Routes startup summary (total:4 started:4)
2021-11-05 12:19:47 INFO AbstractCamelContext:2985 - Started udp-multicast-dispatcher (netty://udp://0.0.0.0:20777)
2021-11-05 12:19:47 INFO AbstractCamelContext:2985 - Started udp-kafka-raw-packets (direct://raw-packets)
2021-11-05 12:19:47 INFO AbstractCamelContext:2985 - Started udp-kafka-events (direct://events)
2021-11-05 12:19:47 INFO AbstractCamelContext:2985 - Started route1 (direct://drivers)
2021-11-05 12:19:47 INFO AbstractCamelContext:2997 - Apache Camel 3.8.0 (camel-1) started in 845ms (build:35ms init:171ms start:639ms)
2021-11-05 12:19:48 INFO Metadata:279 - [Producer clientId=raw-packets] Cluster ID: 9T3oLjNMRw-Gdc6B0nbWAw
2021-11-05 12:19:48 INFO Metadata:279 - [Producer clientId=drivers] Cluster ID: 9T3oLjNMRw-Gdc6B0nbWAw
2021-11-05 12:19:48 INFO Metadata:279 - [Producer clientId=events] Cluster ID: 9T3oLjNMRw-Gdc6B0nbWAw
^C2021-11-05 12:26:06 INFO AbstractCamelContext:3174 - Apache Camel 3.8.0 (camel-1) shutting down (timeout:45s)
2021-11-05 12:26:06 INFO SingleUDPNettyServerBootstrapFactory:201 - ConnectionlessBootstrap disconnecting from 0.0.0.0:20777
2021-11-05 12:26:06 INFO NettyConsumer:86 - Netty consumer unbound from: 0.0.0.0:20777
2021-11-05 12:26:06 INFO KafkaProducer:1205 - [Producer clientId=drivers] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2021-11-05 12:26:06 INFO Metrics:668 - Metrics scheduler closed
2021-11-05 12:26:06 INFO Metrics:672 - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2021-11-05 12:26:06 INFO Metrics:678 - Metrics reporters closed
2021-11-05 12:26:06 INFO AppInfoParser:83 - App info kafka.producer for drivers unregistered
2021-11-05 12:26:06 INFO KafkaProducer:1205 - [Producer clientId=events] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2021-11-05 12:26:06 INFO Metrics:668 - Metrics scheduler closed
2021-11-05 12:26:06 INFO Metrics:672 - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2021-11-05 12:26:06 INFO Metrics:678 - Metrics reporters closed
2021-11-05 12:26:06 INFO AppInfoParser:83 - App info kafka.producer for events unregistered
2021-11-05 12:26:06 INFO KafkaProducer:1205 - [Producer clientId=raw-packets] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2021-11-05 12:26:06 INFO Metrics:668 - Metrics scheduler closed
2021-11-05 12:26:06 INFO Metrics:672 - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2021-11-05 12:26:06 INFO Metrics:678 - Metrics reporters closed
2021-11-05 12:26:06 INFO AppInfoParser:83 - App info kafka.producer for raw-packets unregistered
2021-11-05 12:26:06 INFO AbstractCamelContext:3368 - Routes shutdown summary (total:4 stopped:4)
2021-11-05 12:26:06 INFO AbstractCamelContext:3373 - Stopped route1 (direct://drivers)
2021-11-05 12:26:06 INFO AbstractCamelContext:3373 - Stopped udp-kafka-events (direct://events)
2021-11-05 12:26:06 INFO AbstractCamelContext:3373 - Stopped udp-kafka-raw-packets (direct://raw-packets)
2021-11-05 12:26:06 INFO AbstractCamelContext:3373 - Stopped udp-multicast-dispatcher (netty://udp://0.0.0.0:20777)
2021-11-05 12:26:06 INFO AbstractCamelContext:3313 - Apache Camel 3.8.0 (camel-1) shutdown in 53ms (uptime:6m19s)
I have deployed the manifests in https://github.com/b4mad/formula1-telemetry-kafka/tree/master/manifests via
kustomize build . | oc apply -f -Now I have routes and services
When I try to access the bootstrap server via
KAFKA_BOOTSTRAP_SERVERS=my-cluster-kafka-bootstrap-b4mad-racing.apps.smaug.na.operate-first.cloud:9094 java -jar udp-kafka/target/f1-telemetry-udp-kafka-1.0-SNAPSHOT-jar-with-dependencies.jarI get
How can I expose the bootstrap server (without TLS, for starters)? Or even better, can I run
f1-telemetry-udp-kafka-1.0-SNAPSHOT-jar-with-dependencies.jarinside the cluster and therefore remove the need to run a java app in the local network?