search

spring-projects / spring-kafka

Provides Familiar Spring Abstractions for Apache Kafka
https://projects.spring.io/spring-kafka
Apache License 2.0
2.15k stars 1.54k forks source link

messages duplicated during rebalancing after service recovery from Kafka SSLHandshakeException #2266

Closed vishali0711 closed 2 years ago

vishali0711 commented 2 years ago

Current setup of my spring-boot service - Our Springboot application consumes messages from Kafka topic,We are processing one message at a time (we are not using streams).Below are the config properties and version being used.

ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG- 30000
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG-earliest
ContainerProperties.AckMode-RECORD

Spring boot version-2.5.7 Spring-kafka version- 2.7.8 Kafks-clients version-2.8.1 number of partitions- 6 consumer group- 1 consumers- 2

Issue - When springboot application stays idle for longer time(idle time varying from 4 hrs to 3 days).We are seeing org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Exception error message - org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-2.broker.emh-dev.service.dev found. 2022-04-07 06:58:42.437 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception, retrying in 10000 ms

After service recover we are seeing message duplication with same partition and offsets which is inconsistent.

Below are the exception: Consumer clientId=XXXXXX, groupId=XXXXXX] Offset commit failed on partition XXXXXX at offset 354: The coordinator is not aware of this member

Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records

garyrussell commented 2 years ago

Please provide the complete log between those exceptions.

garyrussell commented 2 years ago

And don't ask the same question in multiple places - it wastes your time and ours.

https://stackoverflow.com/questions/72171153/messages-duplicated-during-rebalancing-after-service-recovery-from-kafka-sslhand

vishali0711 commented 2 years ago

{"instant":{"epochSecond":1651556329,"nanoOfSecond":988000000},"thread":"org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1","level":"ERROR","loggerName":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","message":"Consumer exception","thrown":{"commonElementCount":0,"localizedMessage":"This error handler cannot process 'org.apache.kafka.common.errors.SslAuthenticationException's; no record information is available","message":"This error handler cannot process 'org.apache.kafka.common.errors.SslAuthenticationException's; no record information is available","name":"java.lang.IllegalStateException","cause":{"commonElementCount":0,"localizedMessage":"SSL handshake failed","message":"SSL handshake failed","name":"org.apache.kafka.common.errors.SslAuthenticationException","cause":{"commonElementCount":3,"localizedMessage":"No subject alternative DNS name matching * found.","message":"No subject alternative DNS name matching **** found.","name":"javax.net.ssl.SSLHandshakeException","cause":{"commonElementCount":3,"localizedMessage":"No subject alternative DNS name matching ** found.","message":"No subject alternative DNS name matching * found.","name":"java.security.cert.CertificateException","extendedStackTrace":[{"class":"sun.security.util.HostnameChecker","method":"matchDNS","file":"HostnameChecker.java","line":230,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.util.HostnameChecker","method":"match","file":"HostnameChecker.java","line":106,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.X509TrustManagerImpl","method":"checkIdentity","file":"X509TrustManagerImpl.java","line":457,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.X509TrustManagerImpl","method":"checkIdentity","file":"X509TrustManagerImpl.java","line":417,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.X509TrustManagerImpl","method":"checkTrusted","file":"X509TrustManagerImpl.java","line":285,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.X509TrustManagerImpl","method":"checkServerTrusted","file":"X509TrustManagerImpl.java","line":141,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.CertificateMessage$T12CertificateConsumer","method":"checkServerCerts","file":"CertificateMessage.java","line":632,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.CertificateMessage$T12CertificateConsumer","method":"onCertificate","file":"CertificateMessage.java","line":473,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.CertificateMessage$T12CertificateConsumer","method":"consume","file":"CertificateMessage.java","line":369,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLHandshake","method":"consume","file":"SSLHandshake.java","line":377,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.HandshakeContext","method":"dispatch","file":"HandshakeContext.java","line":444,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction","method":"run","file":"SSLEngineImpl.java","line":981,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction","method":"run","file":"SSLEngineImpl.java","line":968,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"java.security.AccessController","method":"doPrivileged","file":"AccessController.java","line":-2,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLEngineImpl$DelegatedTask","method":"run","file":"SSLEngineImpl.java","line":915,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"runDelegatedTasks","file":"SslTransportLayer.java","line":430,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"handshakeUnwrap","file":"SslTransportLayer.java","line":514,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"doHandshake","file":"SslTransportLayer.java","line":368,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"handshake","file":"SslTransportLayer.java","line":291,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.KafkaChannel","method":"prepare","file":"KafkaChannel.java","line":178,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.Selector","method":"pollSelectionKeys","file":"Selector.java","line":543,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.Selector","method":"poll","file":"Selector.java","line":481,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.NetworkClient","method":"poll","file":"NetworkClient.java","line":561,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":265,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":236,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"pollForFetches","file":"KafkaConsumer.java","line":1296,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"poll","file":"KafkaConsumer.java","line":1237,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"poll","file":"KafkaConsumer.java","line":1210,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"doPoll","file":"KafkaMessageListenerContainer.java","line":1414,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"pollAndInvoke","file":"KafkaMessageListenerContainer.java","line":1251,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"run","file":"KafkaMessageListenerContainer.java","line":1163,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"}]},"extendedStackTrace":[{"class":"sun.security.ssl.Alert","method":"createSSLException","file":"Alert.java","line":131,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.TransportContext","method":"fatal","file":"TransportContext.java","line":324,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.TransportContext","method":"fatal","file":"TransportContext.java","line":267,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.TransportContext","method":"fatal","file":"TransportContext.java","line":262,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.CertificateMessage$T12CertificateConsumer","method":"checkServerCerts","file":"CertificateMessage.java","line":654,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.CertificateMessage$T12CertificateConsumer","method":"onCertificate","file":"CertificateMessage.java","line":473,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.CertificateMessage$T12CertificateConsumer","method":"consume","file":"CertificateMessage.java","line":369,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLHandshake","method":"consume","file":"SSLHandshake.java","line":377,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.HandshakeContext","method":"dispatch","file":"HandshakeContext.java","line":444,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction","method":"run","file":"SSLEngineImpl.java","line":981,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction","method":"run","file":"SSLEngineImpl.java","line":968,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"java.security.AccessController","method":"doPrivileged","file":"AccessController.java","line":-2,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"sun.security.ssl.SSLEngineImpl$DelegatedTask","method":"run","file":"SSLEngineImpl.java","line":915,"exact":false,"location":"?","version":"1.8.0_322"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"runDelegatedTasks","file":"SslTransportLayer.java","line":430,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"handshakeUnwrap","file":"SslTransportLayer.java","line":514,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"doHandshake","file":"SslTransportLayer.java","line":368,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.SslTransportLayer","method":"handshake","file":"SslTransportLayer.java","line":291,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.KafkaChannel","method":"prepare","file":"KafkaChannel.java","line":178,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.Selector","method":"pollSelectionKeys","file":"Selector.java","line":543,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.common.network.Selector","method":"poll","file":"Selector.java","line":481,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.NetworkClient","method":"poll","file":"NetworkClient.java","line":561,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":265,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":236,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"pollForFetches","file":"KafkaConsumer.java","line":1296,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"poll","file":"KafkaConsumer.java","line":1237,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"poll","file":"KafkaConsumer.java","line":1210,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"doPoll","file":"KafkaMessageListenerContainer.java","line":1414,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"pollAndInvoke","file":"KafkaMessageListenerContainer.java","line":1251,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"run","file":"KafkaMessageListenerContainer.java","line":1163,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"}]}},"extendedStackTrace":[{"class":"org.springframework.kafka.listener.SeekUtils","method":"seekOrRecover","file":"SeekUtils.java","line":200,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.SeekToCurrentErrorHandler","method":"handle","file":"SeekToCurrentErrorHandler.java","line":112,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"handleConsumerException","file":"KafkaMessageListenerContainer.java","line":1604,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"run","file":"KafkaMessageListenerContainer.java","line":1212,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"java.util.concurrent.Executors$RunnableAdapter","method":"call","file":"Executors.java","line":511,"exact":true,"location":"?","version":"1.8.0_322"},{"class":"java.util.concurrent.FutureTask","method":"run","file":"FutureTask.java","line":266,"exact":true,"location":"?","version":"1.8.0_322"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":750,"exact":true,"location":"?","version":"1.8.0_322"}]},"endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{},"threadId":58,"threadPriority":5,"source":{"class":"org.springframework.core.log.LogAccessor","method":"error","file":"LogAccessor.java","line":149}}

{"instant":{"epochSecond":1651646141,"nanoOfSecond":222000000},"thread":"org.springframework.kafka.KafkaListenerEndpointContainer#0-4-C-1","level":"ERROR","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","message":"[Consumer clientId=XXXXXXXX, groupId=XXXXXXX] Offset commit failed on partition XXXXXXXXX at offset 312: The coordinator is not aware of this member.","endOfBatch":true,"loggerFqcn":"org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger","contextMap":{},"threadId":70,"threadPriority":5,"source":{"class":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler","method":"handle","file":"ConsumerCoordinator.java","line":1190}}

{"instant":{"epochSecond":1651646142,"nanoOfSecond":340000000},"thread":"org.springframework.kafka.KafkaListenerEndpointContainer#0-4-C-1","level":"ERROR","loggerName":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","message":"Error handler threw an exception","thrown":{"commonElementCount":0,"localizedMessage":"Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.","message":"Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.","name":"org.springframework.kafka.KafkaException","cause":{"commonElementCount":9,"localizedMessage":"Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.","message":"Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.","name":"org.apache.kafka.clients.consumer.CommitFailedException","extendedStackTrace":[{"class":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler","method":"handle","file":"ConsumerCoordinator.java","line":1262,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler","method":"handle","file":"ConsumerCoordinator.java","line":1163,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler","method":"onSuccess","file":"AbstractCoordinator.java","line":1182,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler","method":"onSuccess","file":"AbstractCoordinator.java","line":1157,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.RequestFuture$1","method":"onSuccess","file":"RequestFuture.java","line":206,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.RequestFuture","method":"fireSuccess","file":"RequestFuture.java","line":169,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.RequestFuture","method":"complete","file":"RequestFuture.java","line":129,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler","method":"fireCompletion","file":"ConsumerNetworkClient.java","line":602,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"firePendingCompletedRequests","file":"ConsumerNetworkClient.java","line":412,"exact":false8,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":297,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":236,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient","method":"poll","file":"ConsumerNetworkClient.java","line":215,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","method":"commitOffsetsSync","file":"ConsumerCoordinator.java","line":1005,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.apache.kafka.clients.consumer.KafkaConsumer","method":"commitSync","file":"KafkaConsumer.java","line":1490,"exact":false,"location":"kafka-clients-2.8.1.jar!/","version":"?"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"doCommitSync","file":"KafkaMessageListenerContainer.java","line":2721,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"commitSync","file":"KafkaMessageListenerContainer.java","line":2716,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"ackCurrent","file":"KafkaMessageListenerContainer.java","line":2447,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"invokeOnMessage","file":"KafkaMessageListenerContainer.java","line":2317,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"doInvokeRecordListener","file":"KafkaMessageListenerContainer.java","line":2229,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"}]},"extendedStackTrace":[{"class":"org.springframework.kafka.listener.SeekUtils","method":"seekOrRecover","file":"SeekUtils.java","line":206,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.SeekToCurrentErrorHandler","method":"handle","file":"SeekToCurrentErrorHandler.java","line":112,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"invokeErrorHandler","file":"KafkaMessageListenerContainer.java","line":2371,"exact":false,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"doInvokeRecordListener","file":"KafkaMessageListenerContainer.java","line":2240,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"doInvokeWithRecords","file":"KafkaMessageListenerContainer.java","line":2154,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"invokeRecordListener","file":"KafkaMessageListenerContainer.java","line":2036,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"invokeListener","file":"KafkaMessageListenerContainer.java","line":1709,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"invokeIfHaveRecords","file":"KafkaMessageListenerContainer.java","line":1276,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"pollAndInvoke","file":"KafkaMessageListenerContainer.java","line":1268,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","method":"run","file":"KafkaMessageListenerContainer.java","line":1163,"exact":true,"location":"spring-kafka-2.7.8.jar!/","version":"2.7.8"},{"class":"java.util.concurrent.Executors$RunnableAdapter","method":"call","file":"Executors.java","line":511,"exact":true,"location":"?","version":"1.8.0_322"},{"class":"java.util.concurrent.FutureTask","method":"run","file":"FutureTask.java","line":266,"exact":true,"location":"?","version":"1.8.0_322"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":750,"exact":true,"location":"?","version":"1.8.0_322"}]},"endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{},"threadId":70,"threadPriority":5,"source":{"class":"org.springframework.core.log.LogAccessor","method":"error","file":"LogAccessor.java","line":149}}

vishali0711 commented 2 years ago

please find the log above

garyrussell commented 2 years ago

That is not the complete log; there are many seconds (89812) between those log entries. 

Tue May 03 01:38:49 EDT 2022
Wed May 04 02:35:42 EDT 2022

In order to figure out why the consumer was kicked out of the group, you need to look at the recent logs before the CommitFailedException is thrown.

It is probably completely unrelated to recovery from the SSL problems.

vishali0711 commented 2 years ago

Actually these are the complete logs, in between Tue May 03 01:38:49 EDT 2022 and Wed May 04 02:35:42 EDT 2022,producer did not produce any messages and consumer was idle. We saw commit failed exception once consumer application started consuming the messages

garyrussell commented 2 years ago

Then you don't have sufficient logging levels set to debug such problems, you should at least have INFO logs (which will show rebalance activity).

vishali0711 commented 2 years ago

Hi Gary,

Apologies for the inconvenience caused. Please find below the complete log. These are the logs with higher versions of spring-kafka and kafka-clients but the issue is same.

2022-04-07 06:58:32.379 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception, retrying in 10000 ms

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165) ~[?:1.8.0_201] at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:478) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:341) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1520) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1510) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1338) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) [spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1520) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1510) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1338) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) ~[spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201] Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-2.broker.** found. at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214) ~[?:1.8.0_201] at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1520) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1510) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1338) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) ~[spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201]

2022-04-07 06:58:42.437 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception, retrying in 10000 ms

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165) ~[?:1.8.0_201] at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:478) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:341) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1520) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1510) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1338) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) [spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1520) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1510) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1338) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) ~[spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201] Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-2.broker.** found. at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214) ~[?:1.8.0_201] at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1520) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1510) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1338) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) ~[spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201]

2022-04-07 06:58:52.527 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception, retrying in 10000 ms

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165) ~[?:1.8.0_201] at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:478) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:341) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1386) ~[kafka-clients-3.1.0.jar:?] Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1386) ~[kafka-clients-3.1.0.jar:?] Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-1.broker.** found. at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214) ~[?:1.8.0_201] at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1386) ~[kafka-clients-3.1.0.jar:?]

2022-04-07 06:59:02.594 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception, retrying in 10000 ms

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165) ~[?:1.8.0_201] at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:478) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:341) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1386) ~[kafka-clients-3.1.0.jar:?] Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_201] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1386) ~[kafka-clients-3.1.0.jar:?] Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-1.broker.*** found. at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214) ~[?:1.8.0_201] at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:1.8.0_201] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:1.8.0_201] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_201] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_201] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_201] at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.common.network.Selector.poll(Selector.java:481) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1386) ~[kafka-clients-3.1.0.jar:?]

2022-04-07 10:30:55.929 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 330 2022-04-07 10:31:27.104 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.Sender : Published message 2022-04-07 10:31:27.552 ERROR 24180 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator : [Consumer clientId=consumer-local-1, groupId=consumer-local] Offset commit failed on partition TEST-v1-0 at offset 331: The coordinator is not aware of this member. 2022-04-07 10:31:28.646 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Error handler threw an exception

org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. at org.springframework.kafka.listener.SeekUtils.seekOrRecover(SeekUtils.java:207) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.SeekToCurrentErrorHandler.handle(SeekToCurrentErrorHandler.java:113) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.ErrorHandlerAdapter.handleRemaining(ErrorHandlerAdapter.java:141) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:2666) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2547) [spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427) [spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305) [spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979) [spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364) [spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355) [spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247) [spring-kafka-2.8.4.jar:2.8.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:1286) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:1187) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1196) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1171) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:1029) ~[kafka-clients-3.1.0.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1492) ~[kafka-clients-3.1.0.jar:?] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doCommitSync(KafkaMessageListenerContainer.java:3024) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.commitSync(KafkaMessageListenerContainer.java:3019) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.ackCurrent(KafkaMessageListenerContainer.java:2743) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2611) ~[spring-kafka-2.8.4.jar:2.8.4] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536) ~[spring-kafka-2.8.4.jar:2.8.4] ... 9 more

2022-04-07 10:31:33.733 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 330 2022-04-07 10:31:34.255 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.Sender : Published message 2022-04-07 10:31:48.262 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 331 2022-04-07 10:31:48.823 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.Sender : Published message to solace 2022-04-07 10:31:56.101 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 332 2022-04-07 10:31:56.101 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Constructed safestore object with id 721a26f8-628d-4a5e-8eb4-e16a037c809b 2022-04-07 10:31:56.603 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.Sender : Published message 2022-04-07 10:32:03.666 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 333 2022-04-07 10:32:04.155 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.Sender : Published message 2022-04-07 10:32:11.261 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 334 2022-04-07 10:32:11.773 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.Sender : Published message 2022-04-07 10:32:18.454 INFO 24180 --- [ntainer#0-0-C-1] c.a.p.d.m.ConsumerTopology : Received Message from kafka topic TEST-v1 with record key t409:4bd335e1-6f59-4f70-8840-cdc53ef36ec07:0900 partition 0 offset 335

if any more logs are needed apart from these please let us know what level of logging we can enable.

Thanks.

vishali0711 commented 2 years ago

We are seeing duplicate messages around commit failure.Sometimes we are seeing duplicates just before commit failure,sometimes message is received then commit failed Exception is thrown and then same message being consumed again. for example here for offset 330 we can see duplicates. Appreciate your help on this.

vishali0711 commented 2 years ago

@garyrussell can you please help on this

garyrussell commented 2 years ago

Sorry for the delay; I don't work on Fridays.

It's clearly not related to the SSL/auth issues.

The commit error means your listener is taking too long to process the records returned from the previous poll and a rebalance has occurred. Hence the record is redelivered.

You must set max.poll.records and max.poll.interval.ms such that you will never take longer than that to process the records received.

vishali0711 commented 2 years ago

Hi @garyrussell thanks for the details, we have reduced max.poll.records from default value to 50, but this did not help,we are seeing same commit failed issue.Please suggest.

artembilan commented 2 years ago

Have you tried to increase max.poll.interval.ms then? As you see the rebalance on the consumer group happens when we process a records from the current poll longer than it is allowed by consumer activity algorithm.