search

camunda / camunda

Process Orchestration Framework
https://camunda.com/platform/
3.35k stars 612 forks source link

Camunda Exporter fails with: Failed retrieving mappings from index/index templates with pattern [*] #24243

Open ChrisKujawa opened 4 weeks ago

ChrisKujawa commented 4 weeks ago

Description

Reported on INT with our RC for alpha1

https://console.cloud.google.com/errors/detail/CI7_od-oyLDzSA;service=zeebe;time=P7D;locations=global?project=camunda-saas-int

io.camunda.exporter.exceptions.ElasticsearchExporterException: Failed retrieving mappings from index/index templates with pattern [*]

at io.camunda.exporter.schema.elasticsearch.ElasticsearchEngineClient.getMappings ( io/camunda.exporter.schema.elasticsearch/ElasticsearchEngineClient.java:131 )
at io.camunda.exporter.schema.SchemaManager.validateIndexTemplates ( io/camunda.exporter.schema/SchemaManager.java:161 )
at io.camunda.exporter.schema.SchemaManager.startup ( io/camunda.exporter.schema/SchemaManager.java:47 )
at io.camunda.exporter.CamundaExporter.open ( io/camunda.exporter/CamundaExporter.java:102 )
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$openExporter$0 ( io/camunda.zeebe.broker.exporter.stream/ExporterContainer.java:118 )
at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader ( io/camunda.zeebe.util.jar/ThreadContextUtil.java:59 )
at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader ( io/camunda.zeebe.util.jar/ThreadContextUtil.java:35 )
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.openExporter ( io/camunda.zeebe.broker.exporter.stream/ExporterContainer.java:117 )
at io.camunda.zeebe.broker.exporter.stream.ExporterDirector.lambda$startActiveExportingMode$16 ( io/camunda.zeebe.broker.exporter.stream/ExporterDirector.java:532 )
at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run ( io/camunda.zeebe.scheduler.retry/BackOffRetryStrategy.java:51 )
at io.camunda.zeebe.scheduler.ActorJob.invoke ( io/camunda.zeebe.scheduler/ActorJob.java:85 )
at io.camunda.zeebe.scheduler.ActorJob.execute ( io/camunda.zeebe.scheduler/ActorJob.java:42 )
at io.camunda.zeebe.scheduler.ActorTask.execute ( io/camunda.zeebe.scheduler/ActorTask.java:122 )
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask ( io/camunda.zeebe.scheduler/ActorThread.java:130 )
at io.camunda.zeebe.scheduler.ActorThread.doWork ( io/camunda.zeebe.scheduler/ActorThread.java:108 )
at io.camunda.zeebe.scheduler.ActorThread.run ( io/camunda.zeebe.scheduler/ActorThread.java:227 )
Caused by: co.elastic.clients.transport.TransportException

…
at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse ( ElasticsearchTransportBase.java:404 )
at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse ( ElasticsearchTransportBase.java:363 )
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest ( ElasticsearchTransportBase.java:147 )
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getIndexTemplate ( ElasticsearchIndicesClient.java:1163 )
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getIndexTemplate ( ElasticsearchIndicesClient.java:1180 )
at io.camunda.exporter.schema.elasticsearch.ElasticsearchEngineClient.getCurrentMappings ( ElasticsearchEngineClient.java:204 )
at io.camunda.exporter.schema.elasticsearch.ElasticsearchEngineClient.getMappings ( ElasticsearchEngineClient.java:114 )
…
Caused by: co.elastic.clients.json.JsonpMappingException

…
at co.elastic.clients.json.JsonpUtils.lookAheadFieldValue ( JsonpUtils.java:239 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:201 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.BuildFunctionDeserializer.deserialize ( BuildFunctionDeserializer.java:47 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize ( JsonpDeserializerBase.java:349 )
at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize ( JsonpDeserializerBase.java:333 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:79 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:79 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:79 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:79 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:79 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:85 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:48 )
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize ( JsonpDeserializerBase.java:318 )
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize ( JsonpDeserializerBase.java:280 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize ( ObjectDeserializer.java:78 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:192 )
at co.elastic.clients.json.ObjectDeserializer.deserialize ( ObjectDeserializer.java:148 )
at co.elastic.clients.json.JsonpDeserializer.deserialize ( JsonpDeserializer.java:77 )
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize ( ObjectBuilderDeserializer.java:79 )
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize ( DelegatingDeserializer.java:43 )
at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse ( ElasticsearchTransportBase.java:399 )
at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse ( ElasticsearchTransportBase.java:363 )
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest ( ElasticsearchTransportBase.java:147 )
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getIndexTemplate ( ElasticsearchIndicesClient.java:1163 )
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getIndexTemplate ( ElasticsearchIndicesClient.java:1180 )
at io.camunda.exporter.schema.elasticsearch.ElasticsearchEngineClient.getCurrentMappings ( ElasticsearchEngineClient.java:204 )
at io.camunda.exporter.schema.elasticsearch.ElasticsearchEngineClient.getMappings ( ElasticsearchEngineClient.java:114 )
…
EuroLew commented 4 weeks ago

There's an issue deserializing index template 95, saying there is no type field. Elasticsearch 7.x does not need a type field but as per the 8.15 documentation here a type field is needed for a lowercase normalizer.

Given we use the 8.x version of the elasticsearch-java client when deserializing it expects the type field.

In operate-list-view.json I updated this field to have the type

image

I think the issue is this.

  1. we have a new json file for operate-list-view
  2. schema start up occurs.
  3. before we can go an update the operate list view template we validate the templates
  4. when validating the templates if verifies the state in elasticsearch against the json file we have, to do this it must retrieve the template from elasticsearch and the deserialise so it can be compared
  5. it is at this deserialisation where the failure occurs

Possible fixes: Depending on how quick this is needed to be alleviated

ChrisKujawa commented 4 weeks ago

@EuroLew couldn't the conflict come from the Operate Importer?

EuroLew commented 4 weeks ago

If we are starting fresh with an empty elasticsearch this issue shouldn't occur as we would create the template correctly. This only occurs if an old version of the template exists I assumed it was the case since some of these errored namespaces are over a month old that a. we had a camunda instance, we turned on the exporter (This would error as the importer created the indices and theres a conflict) Or b. If we start from fresh with importer and exporter then yes there is a race condition that could cause this if the importer gets to create the index first

ChrisKujawa commented 4 weeks ago

We likely should resolve this via https://github.com/camunda/camunda/issues/22897

ChrisKujawa commented 4 weeks ago

@sdorokhova could we already fix this https://github.com/camunda/camunda/issues/22897 so we resolve this issue here?