search

EC-Release / sdk

The Agent SDK
Other
4 stars 7 forks source link

Unable to establish connectivity with AWS MSK from Onprem through EC #130

Closed Prakash-droid closed 4 years ago

Prakash-droid commented 4 years ago

We are facing an issue (ssl error in case of tls connectivity) while establishing the connectivity from on-prem to msk(kafka) cluster running as a service in AWS. Kindly help

Details from App Logs: [2020-08-13 07:36:56,410] ERROR [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:7989) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2020-08-13 07:36:56,411] WARN [Producer clientId=console-producer] Bootstrap broker localhost:7989 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

ayasuda2OO3 commented 4 years ago

Is there a EC component(s) deployed in a k8s cluster? @Prakash-droid

Prakash-droid commented 4 years ago

Nope, We are not using the one from K8S cluster. Server agent is on one of the EC2 sandbox.

Prakash-droid commented 4 years ago

Additional Details on the components: Source : alpcclappdvn01.corporate.ge.com(OnPrem) Target : b-2.dc-kafka-poc-cloudwatc.ajz1qp.c1.kafka.us-east-1.amazonaws.com:9094(MSK) EC Client on alpcclappdvn01.corporate.ge.com EC Server on EC2 machine (10.242.49.20) (within the same VPC where AWS MSK service is running) TLS Version [Rev:v1.hokkaido.212.tls] EC Agent Version [v1.hokkaido.212]

ayasuda2OO3 commented 4 years ago

Upon the review of the usage, we notice that the kafka client producer is coerced to establish a TLS connection with the client agent (http://localhost:7989, http://127.0.0.1:7989)

The client agent which deployed locally, is by default via the HTTP (Non-TLS) handshake. Please configure/disable your TLS usage in the kafka producer client end to mitigate the connectivity issue.

Reference: https://github.com/EC-Release/sdk/tree/v1.1/plugins/tls

@Prakash-droid @EC-Release/contributors @EC-Release/stakeholders @EC-Release/research-development

Prakash-droid commented 4 years ago

ScriptandLogInfo.txt

Prakash-droid commented 4 years ago

We are trying to connect to 9092 (PLAINTEXT). I have captured the logs and attached to the ticket.Please verify.

./kafka-console-producer.sh --broker-list localhost:7989 --topic corp_item_topic [2020-08-17 13:35:07,831] INFO ProducerConfig values: acks = 1 batch.size = 16384 bootstrap.servers = [localhost:7989] buffer.memory = 33554432 client.dns.lookup = default client.id = console-producer security.protocol = PLAINTEXT

Gayatri212 commented 4 years ago

After disabling the TLS following are the client application logs

/kafka-console-producer.sh --broker-list localhost:7989 --topic corp_item_topic
/usr/bin/java
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
[2020-08-17 12:44:26,770] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-08-17 12:44:26,948] INFO ProducerConfig values:
        acks = 1
        batch.size = 16384
        bootstrap.servers = [localhost:7989]
        buffer.memory = 33554432
        client.dns.lookup = default
        client.id = console-producer
        compression.type = none
        connections.max.idle.ms = 540000
        delivery.timeout.ms = 120000
        enable.idempotence = false
        interceptor.classes = []
        key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
        linger.ms = 1000
        max.block.ms = 60000
        max.in.flight.requests.per.connection = 5
        max.request.size = 1048576
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
        receive.buffer.bytes = 32768
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 1500
        retries = 3
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        security.providers = null
        send.buffer.bytes = 102400
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        transaction.timeout.ms = 60000
        transactional.id = null
        value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
 (org.apache.kafka.clients.producer.ProducerConfig)
[2020-08-17 12:44:27,059] DEBUG [Producer clientId=console-producer] Starting Kafka producer I/O thread. (org.apache.kafka.clients.producer.internals.Sender)
[2020-08-17 12:44:27,063] DEBUG [Producer clientId=console-producer] Initialize connection to node localhost:7989 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:27,065] DEBUG [Producer clientId=console-producer] Initiating connection to node localhost:7989 (id: -1 rack: null) using address localhost/127.0.0.1 (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:27,071] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-08-17 12:44:27,072] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-08-17 12:44:27,072] INFO Kafka startTimeMs: 1597648467056 (org.apache.kafka.common.utils.AppInfoParser)
[2020-08-17 12:44:27,074] DEBUG [Producer clientId=console-producer] Kafka producer started (org.apache.kafka.clients.producer.KafkaProducer)
>[2020-08-17 12:44:27,126] DEBUG [Producer clientId=console-producer] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 102400, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
[2020-08-17 12:44:27,372] DEBUG [Producer clientId=console-producer] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:27,373] DEBUG [Producer clientId=console-producer] Initiating API versions fetch from node -1. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:28,960] DEBUG [Producer clientId=console-producer] Disconnecting from node -1 due to request timeout. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:28,961] WARN [Producer clientId=console-producer] Bootstrap broker localhost:7989 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:29,061] DEBUG [Producer clientId=console-producer] Initialize connection to node localhost:7989 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:29,061] DEBUG [Producer clientId=console-producer] Initiating connection to node localhost:7989 (id: -1 rack: null) using address localhost/127.0.0.1 (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:29,063] DEBUG [Producer clientId=console-producer] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 102400, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
[2020-08-17 12:44:29,063] DEBUG [Producer clientId=console-producer] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:29,063] DEBUG [Producer clientId=console-producer] Initiating API versions fetch from node -1. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,565] DEBUG [Producer clientId=console-producer] Disconnecting from node -1 due to request timeout. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,565] WARN [Producer clientId=console-producer] Bootstrap broker localhost:7989 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,665] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,716] DEBUG [Producer clientId=console-producer] Initialize connection to node localhost:7989 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,716] DEBUG [Producer clientId=console-producer] Initiating connection to node localhost:7989 (id: -1 rack: null) using address localhost/127.0.0.1 (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,727] DEBUG [Producer clientId=console-producer] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 102400, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
[2020-08-17 12:44:30,727] DEBUG [Producer clientId=console-producer] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:30,727] DEBUG [Producer clientId=console-producer] Initiating API versions fetch from node -1. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:32,229] DEBUG [Producer clientId=console-producer] Disconnecting from node -1 due to request timeout. (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:32,229] WARN [Producer clientId=console-producer] Bootstrap broker localhost:7989 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:32,328] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2020-08-17 12:44:32,380] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.N
ayasuda2OO3 commented 4 years ago

@Prakash-droid @Gayatri212

Could you share the server log? a snippet of the latest connectivity should be just fine.

Gayatri212 commented 4 years ago

This issue is resolved by following action

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Gayatri212 commented 4 years ago

The issue is resolved