source connector elasticsearch failed

[SarindraTherese]

Hey, I have some error in log when i try to use elasticsearch source connector

[2022-08-19 11:53:46,179] ERROR [elastic-source|worker] error in searching index names (com.github.dariobalinzo.elastic.ElasticRepository:193)
[2022-08-19 11:53:46,180] ERROR [elastic-source|worker] Error while trying to get updated topics list, ignoring and waiting for next table poll interval (com.github.dariobalinzo.elastic.ElasticIndexMonitorThread:92)

this link https://localhost:9200/_cat/indices?v return

health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   kibana_sample_data_ecommerce tRcxyLrsSKuWcmOx3ZNKew   1   0       4675            0        4mb            4mb
yellow open   index-elastic                CvedHZsxTB6aGeP9cUVedw   1   1          0            0       225b           225b
yellow open   test                         R9t_ToRpQGyi_tTw6MdkSA   1   1         15            0     15.5kb         15.5kb

And this is my properties connector:

{
  "name": "elastic-source",
  "config": {
    "name": "elastic-source",
    "connector.class": "com.github.dariobalinzo.ElasticSourceConnector",
    "tasks.max": "1",
    "errors.log.enable": "true",
    "errors.log.include.messages": "true",
    "es.host": "127.0.0.1",
    "es.port": "9200",
    "es.user": "elastic",
    "es.password": "password",
    "index.prefix": "test",
    "topic.prefix": "test"
  }
}

As we see, text index is already exist, so what's wrong with my approach?

[DarioBalinzo]

Hi, thanks for reporting this.

Can you share the full stacktrace below 2022-08-19 11:53:46,180] ERROR [elastic-source|worker] Error while trying to get updated topics list, ignoring and waiting for next table poll interval (com.github.dariobalinzo.elastic.ElasticIndexMonitorThread:92) ?

[SarindraTherese]

Yes!

(org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:376)
[2022-08-19 15:18:47,623] ERROR [elastic-source|worker] error in searching index names (com.github.dariobalinzo.elastic.ElasticRepository:193)
[2022-08-19 15:18:47,624] ERROR [elastic-source|worker] Error while trying to get updated topics list, ignoring and waiting for next table poll interval (com.github.dariobalinzo.elastic.ElasticIndexMonitorThread:92)
java.lang.RuntimeException: org.apache.http.ConnectionClosedException: Connection is closed
    at com.github.dariobalinzo.elastic.ElasticRepository.catIndices(ElasticRepository.java:194)
    at com.github.dariobalinzo.elastic.ElasticIndexMonitorThread.updateIndexes(ElasticIndexMonitorThread.java:89)
    at com.github.dariobalinzo.elastic.ElasticIndexMonitorThread.run(ElasticIndexMonitorThread.java:45)
Caused by: org.apache.http.ConnectionClosedException: Connection is closed
    at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:886)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:276)
    at com.github.dariobalinzo.elastic.ElasticRepository.catIndices(ElasticRepository.java:191)
    ... 2 more
Caused by: org.apache.http.ConnectionClosedException: Connection is closed
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:829)

[SarindraTherese]

And this is in above

org.apache.kafka.connect.errors.ConnectException: Cannot find any elasticsearch index
    at com.github.dariobalinzo.elastic.ElasticIndexMonitorThread.indexes(ElasticIndexMonitorThread.java:77)
    at com.github.dariobalinzo.ElasticSourceConnector.findTaskFromIndexPrefix(ElasticSourceConnector.java:131)
    at com.github.dariobalinzo.ElasticSourceConnector.taskConfigs(ElasticSourceConnector.java:121)
    at org.apache.kafka.connect.runtime.Worker.connectorTaskConfigs(Worker.java:383)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnector(DistributedHerder.java:1574)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnectorTasksWithRetry(DistributedHerder.java:1521)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$null$30(DistributedHerder.java:1534)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:408)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:326)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
[2022-08-19 17:59:08,696] INFO SourceConnectorConfig values: 
    config.action.reload = restart
    connector.class = com.github.dariobalinzo.ElasticSourceConnector
    errors.log.enable = true
    errors.log.include.messages = true
    errors.retry.delay.max.ms = 60000
    errors.retry.timeout = 0
    errors.tolerance = none
    header.converter = null
    key.converter = class org.apache.kafka.connect.json.JsonConverter
    name = elastic-source
    predicates = []
    tasks.max = 1
    topic.creation.groups = []
    transforms = []
    value.converter = class org.apache.kafka.connect.json.JsonConverter
 (org.apache.kafka.connect.runtime.SourceConnectorConfig:376)
[2022-08-19 17:59:08,696] INFO EnrichedConnectorConfig values: 
    config.action.reload = restart
    connector.class = com.github.dariobalinzo.ElasticSourceConnector
    errors.log.enable = true
    errors.log.include.messages = true
    errors.retry.delay.max.ms = 60000
    errors.retry.timeout = 0
    errors.tolerance = none
    header.converter = null
    key.converter = class org.apache.kafka.connect.json.JsonConverter
    name = elastic-source
    predicates = []
    tasks.max = 1
    topic.creation.groups = []
    transforms = []
    value.converter = class org.apache.kafka.connect.json.JsonConverter
 (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:376)

[DarioBalinzo]

Cannot find any elasticsearch index maybe the index was created too late (i.e. the connector was already running)?

can you please try to destroy and recreate it?

Also, ConnectionClosedException: Connection is closed seems to be a network issue

[SarindraTherese]

Maybe the problem is the https of elasticsearch, because I've already try to connect elasticsearch in python but this return error access ssl, so I'm going to fix ssl access about elasticsearch today, therefor if you have some link that helped so i'm interested thanks

[SarindraTherese]

Indeed, it is because of the url https, so I turn false some value in file elasticsearch.yml and the status change to running :

xpack.security.enabled: false
xpack.security.enrollment.enabled: false
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: false

However, I don't know how to solve this case with the https address Thanks

[DarioBalinzo]

closing since the issue seems not related to the connector