QUESTION: So what config would I use to simply pass through SASL?

[joncourt]

Given the answer to a previous question - that SASL will get passed through by default - does this mean that i configure my java client with the SASL properties for the brokers and it will get passed transparently through the kafka proxy when i use the kafka-proxy as my bootstrap-servers?

I've been fiddling around with the settings to achieve this and i'm up to the point where i'm getting net address mapping not faund for my brokers which appear to have been renamed to b2- (i.e. b2-pkc-ep0ml.ap-southeast-2.aws.confluent.cloud:9092 for the config below). Do i need to configure something in external-server-mappings perhaps - i don't actually understand what external-server-mappings actually does - I originally thought it might be instead of bootstrap-server-mapping where the kafka broker is on a host that's not the same as the one the kafka-proxy is on, but that doesn't appear to be so.

I'm presuming some kind of dynamic proxy is set up on the kafka-proxy and the path is client -> kafka-proxy -> kafka-proxy dynamic proxy -> kafka-brokers?

I think perhaps I'm a little confused.

Background

my java client is configured:

ssl.truststore.location=dev-kafka.truststore.jks
ssl.truststore.password=changeit
ssl.truststore.type=PKCS12
ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
request.timeout.ms=20000
bootstrap.servers=worker-0.some.domain.name.com:9092
retry.backoff.ms=500
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="someusername" password\="somepassword";
security.protocol=SASL_SSL

and my kafka-proxy is run as:

kafka-proxy server \
    --bootstrap-server-mapping "pkc-ep0ml.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9092" \
    --tls-enable \
    --tls-insecure-skip-verify \
    --proxy-listener-tls-enable \
    --proxy-listener-key-file=/root/dev-kafka-server.private.key \
    --proxy-listener-cert-file=/root/dev-kafka-server.signed.cer \
    --dynamic-listeners-disable

Error's I'm seeing are:

INFO[2018-10-08T17:17:51+13:00] New connection for pkc-ep0ml.ap-southeast-2.aws.confluent.cloud:9092 
INFO[2018-10-08T17:17:51+13:00] Reading data from pkc-ep0ml.ap-southeast-2.aws.confluent.cloud:9092 had error: net address mapping for b2-pkc-ep0ml.ap-southeast-2.aws.confluent.cloud:9092 was not found 

[joncourt]

actually I might have a handle on this now.... will get back.

[everesio]

I assume that b2-pkc-ep0ml.ap-southeast-2.aws.confluent.cloud:9092 is an advertised listener sent by the broker.

external-server-mapping is for changing received host:port addresses without starting additional listener. I used this configuration for 1 Proxy to 1 Broker setup e.g.

{{ range $i := until (int $.brokers) }}
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
   name: kafka-gateway-{{ $i }}
   namespace: kafka-gateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-gateway-{{ $i }}
  template:
    metadata:
      labels:
        app: kafka-gateway-{{ $i }}
    spec:
      containers:
        - name: kafka-gateway
          image: grepplabs/kafka-proxy:latest
          imagePullPolicy: Always
          args:
            - 'server'
            - '--log-format=json'
            - '--dynamic-listeners-disable'
            - '--bootstrap-server-mapping=kafka-{{ $i }}-0:9093,0.0.0.0:32400,kafka-gateway-{{ $i }}-0.{{ $.dnszone }}:32400'
            {{- range $j := until (int $.brokers) }}
            - '--external-server-mapping=kafka-{{ $j }}-0:9093,kafka-gateway-{{ $j }}-0.{{ $.dnszone }}:32400'
            {{- end }}
...

[joncourt]

thanks @everesio - i've got it going now. This tool is legend by the way - all going well in terms of performance or similar this tool essentially makes it possible for us to use confluent cloud from our internal corp network!! I'll suggest this as a tool for Confluent themselves to perhaps contribute to if you're ok with that. It'll help them with their story for corporates with annoying networking policies.

this is how I've got it going:

The b2-* is an advertised listener (thanks for the clarification - i had thought the proxy was creating it) - once i realise this it started to make more sense. Another paradigm shift.

So CCloud has a bunch of brokers obviously but they only publish the load balancer for them, obviously tools using the published endpoint get the other brokers back in the meta and link them all up. CCloud is publishing 4 on this dev network.

My configuration below seems to be working well. Note that i've mirrored the ccloud setup by having the kafka-proxy deployed on a matching number of brokers.

kafka-proxy server \
    --bootstrap-server-mapping "our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9092,kafka.our.corp.domain.com:9092" \
    --external-server-mapping "b0-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,kafka-0.dev.our.corp.domain.com:9092" \
    --external-server-mapping "b1-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,kafka-1.dev.our.corp.domain.com:9092" \
    --external-server-mapping "b2-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,kafka-2.dev.our.corp.domain.com:9092" \
    --external-server-mapping "b3-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,kafka-3.dev.our.corp.domain.com:9092" \
    --tls-enable \
    --proxy-listener-tls-enable \
    --proxy-listener-key-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-key.pem \
    --proxy-listener-cert-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-cert.pem \
    --dynamic-listeners-disable \
    --default-listener-ip 10.209.59.32 

Your comments on whether this makes sense are welcome please?

[joncourt]

hmm... there does seem to be a problem with the all but one nodes, the others are getting back node ip's of 0.0.0.0 and failing. the one that is working is returning the local fqdn - kafka-0.dev.... etc.

actually looks like timeouts on 2 of the nodes, the others are ok. investigating tomorrow.

[everesio]

Probably you need to proxy all brokers. If client connects to unproxied broker and it receives Metadata or FindCoordinator than the client will have connection problem as the payload will contain original broker addresses.

With one proxy and if host with 0.0.0.0 binds is the kafka.our.corp.domain.com you could try:

kafka-proxy server \
    --bootstrap-server-mapping "our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9092,kafka.our.corp.domain.com:9092" \
    --bootstrap-server-mapping "b0-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9093,kafka.our.corp.domain.com:9093" \
    --bootstrap-server-mapping "b1-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9094,kafka.our.corp.domain.com:9094" \
    --bootstrap-server-mapping "b2-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9095,kafka.our.corp.domain.com:9095" \
    --bootstrap-server-mapping "b3-our-ccloud-env.ap-southeast-2.aws.confluent.cloud:9092,0.0.0.0:9096,kafka.our.corp.domain.com:9096" \
...

With many proxies e.g. 1 proxy for 1 broker you can use similar config I provided previously.

BTW. 3rd position e.g. (kafka.our.corp.domain.com:9096) is optional and it is similar to Kafka advertised listener, but this is a network address how the proxy can be reached.

[joncourt]

Thanks @everesio. I've actually done something similar to that by having 4 servers (one per ccloud broker) each with their own kafka-proxy instance and where i've tried either:

1. setting the --bootstrap-server-mapping to the load balanced endpoint of ccloud and having all the brokers as --external-server-mappings and rotating the --default-listener-ip as per the host the kafka-proxy is on

or

2. leaving the load balanced endpoint out of it all together and simply rotating the --bootstrap-server-mapping per server, leaving the others as external server mappings (also rotating the -default-listener-ip)

the advertised host particle of bootstrap-server-mapping i set to kafka-x.our.corp.domain.com:9092 where x is the node number of 0 through 3.

either seem to be work sort of ok and rotating through hall the brokers regardless of what server i hit first from my laptop - i'm finding 2 nodes are working great, but the other 2 are getting timeout issues like the following when i use them to bootstrap from my laptop:

[2018-10-11 22:51:37,456] ERROR Error when sending message to topic jon_test with key: null, value: 7 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for jon_test-0: 1506 ms has passed since batch creation plus linger time
timeout

which i'm planning to look into over the weekend or early next week. I suspect this particular issue may be i've not done those 2 nodes correctly obviously. Any hints about this one would be appreciated though if it's something you're familiar with.

[joncourt]

Looks like option 2 is the go. It makes sense actually. In hindsight option 1 will likely get in a circular mess frequently if i have all the brokers going to a load balancer downstream.

Config ends up looking like this:

# Broker 0
kafka-proxy server \
  --bootstrap-server-mapping "b0-our-ccloud.region.aws.confluent.cloud:9092,0.0.0.0:9092,kafka-0.our.corp.domain.com:9092" \
  --external-server-mapping "b1-our-ccloud.region.aws.confluent.cloud:9092,kafka-1.our.corp.domain.com:9092" \
  --external-server-mapping "b2-our-ccloud.region.aws.confluent.cloud:9092,kafka-2.our.corp.domain.com:9092" \
  --external-server-mapping "b3-our-ccloud.region.aws.confluent.cloud:9092,kafka-3.our.corp.domain.com:9092" \
  --tls-enable \
  --tls-insecure-skip-verify \
  --proxy-listener-tls-enable \
  --proxy-listener-key-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-key.pem \
  --proxy-listener-cert-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-cert.pem \
  --dynamic-listeners-disable \
  --default-listener-ip 10.209.58.146

# Broker 1
kafka-proxy server \
  --external-server-mapping "b0-our-ccloud.region.aws.confluent.cloud:9092,kafka-0.our.corp.domain.com:9092" \
  --bootstrap-server-mapping "b1-our-ccloud.region.aws.confluent.cloud:9092,0.0.0.0:9092,kafka-1.our.corp.domain.com:9092" \
  --external-server-mapping "b2-our-ccloud.region.aws.confluent.cloud:9092,kafka-2.our.corp.domain.com:9092" \
  --external-server-mapping "b3-our-ccloud.region.aws.confluent.cloud:9092,kafka-3.our.corp.domain.com:9092" \
  --tls-enable \
  --tls-insecure-skip-verify \
  --proxy-listener-tls-enable \
  --proxy-listener-key-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-key.pem \
  --proxy-listener-cert-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-cert.pem \
  --dynamic-listeners-disable \
  --default-listener-ip 10.209.59.32

# Broker 2
kafka-proxy server \
  --external-server-mapping "b0-our-ccloud.region.aws.confluent.cloud:9092,kafka-0.our.corp.domain.com:9092" \
  --external-server-mapping "b1-our-ccloud.region.aws.confluent.cloud:9092,kafka-1.our.corp.domain.com:9092" \
  --bootstrap-server-mapping "b2-our-ccloud.region.aws.confluent.cloud:9092,0.0.0.0:9092,kafka-2.our.corp.domain.com:9092" \
  --external-server-mapping "b3-our-ccloud.region.aws.confluent.cloud:9092,kafka-3.our.corp.domain.com:9092" \
  --tls-enable \
  --tls-insecure-skip-verify \
  --proxy-listener-tls-enable \
  --proxy-listener-key-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-key.pem \
  --proxy-listener-cert-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-cert.pem \
  --dynamic-listeners-disable \
  --default-listener-ip 10.209.58.169

# Broker 3
kafka-proxy server \
  --external-server-mapping "b0-our-ccloud.region.aws.confluent.cloud:9092,kafka-0.our.corp.domain.com:9092" \
  --external-server-mapping "b1-our-ccloud.region.aws.confluent.cloud:9092,kafka-1.our.corp.domain.com:9092" \
  --external-server-mapping "b2-our-ccloud.region.aws.confluent.cloud:9092,kafka-2.our.corp.domain.com:9092" \
  --bootstrap-server-mapping "b3-our-ccloud.region.aws.confluent.cloud:9092,0.0.0.0:9092,kafka-3.our.corp.domain.com:9092" \
  --tls-enable \
  --tls-insecure-skip-verify \
  --proxy-listener-tls-enable \
  --proxy-listener-key-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-key.pem \
  --proxy-listener-cert-file /var/private/ssl/kafka-broker-proxy/kafka-broker-proxy-server-cert.pem \
  --dynamic-listeners-disable \
  --default-listener-ip 10.209.59.54

[everesio]

This is similar configuration is used for kafka-gateway :-) BTW. If a broker is mapped both in the bootstrap-server-mapping and in external-server-mapping, the bootstrap-server-mapping will be used. Such duplication can make the templating easier.

[joncourt]

Legend!!

Thanks for your help @everesio.

[dberardo-com]

i have setup the proxy in front of a single broker, that is advertising an IP which is not known inside the VPN of broker and proxy. To do this i am using a length 3 array like: --bootstrap-server-mapping "b2-our-ccloud.region.aws.confluent.cloud:9092,0.0.0.0:9092,kafka-2.our.corp.domain.com:9092"

the mapping seems to work fine if i try to access the broker from inside the proxy via kafkacat -b localhost:proxy_port -L, but if i try to bind kafkacat to the proxy using the advertised IP i get a strange SASL error, which is not there if i try to open a direct connection to the broker instead.

all listeners are in fact bound using plaintext and use no SASL auth whatsoever.

so basically:

1, broker listening on ports 9092 (internal) and 9093 (external advertised IP), (both plaintext)

2. proxy listening on port 888, binding to 9093 --> broker:9093,0.0.0.0:900,external_IP:888 --> this container has another tcp proxy in front, which publishes port 900 to 888 on the host with external_IP

tests:

1. kcat -b broker:9092 -L --> success
2. kcat -b localhost:900 -L --> success (from inside broker)
3. kcat -b external_IP:888 -L --> fails with SASL exception

is my exception normal?