search

openzipkin / zipkin-dependencies

Spark job that aggregates zipkin spans for use in the UI
Apache License 2.0
176 stars 81 forks source link

deps: updates to latest versions and testcontainers jupiter #220

Closed codefromthecrypt closed 7 months ago

codefromthecrypt commented 7 months ago

This updates to latest dependency versions and migrates tests to use testcontainers jupiter. This also removes use of nested tests as it is simpler to run and diagnose top-level tests. Finally, this fixes logging by using the correct log configuration file for log4j2.

codefromthecrypt commented 7 months ago

elasticsearch job hung.. will look at logs

codefromthecrypt commented 7 months ago

looks like the docker image ran out of memory

codefromthecrypt commented 7 months ago

https://github.com/openzipkin/zipkin/pull/3706 and I'll recut the docker image after it and try again

codefromthecrypt commented 7 months ago

I fixed logging as it was log4j 1.2 and the actual logger is 2, then I refactored the tests to run the heavy stuff in a separate container, and the elasticsearch tests with only one index. crossing fingers it passes.

codefromthecrypt commented 7 months ago

in a bit, new docker images will be cut and we can try again. I have to call it quits as borrowed an entire day for this patch release, and that wasn't time I had.

@reta if you can debug and/or take this forward, much oblidged. You can revert whatever isn't helpful. Basically elasticsearch is timing out on not even difficult work. On my laptop ES 8 times out much more often than 7 for the dependencies job, and the sort of errors you get tend to show up in error searches for a decade.

I would not be opposed to changing this out to opensearch or something else either, but anyway fresh eyes might find a way forward I didn't.

codefromthecrypt commented 7 months ago

building local from https://github.com/openzipkin/zipkin/pull/3708 like so..

DOCKER_FILE=docker/test-images/zipkin-elasticsearch8/Dockerfile build-bin/docker/docker_build openzipkin/zipkin-elasticsearch8:test

with this diff

--- a/elasticsearch/src/main/java/zipkin2/dependencies/elasticsearch/ElasticsearchDependenciesJob.java
+++ b/elasticsearch/src/main/java/zipkin2/dependencies/elasticsearch/ElasticsearchDependenciesJob.java
@@ -135,7 +135,7 @@ public final class ElasticsearchDependenciesJob {
       return this;
     }

-    /** Ensures that logging is setup. Particularly important when in cluster mode. */
+    /** Ensures that logging is set up. Particularly important when in cluster mode. */
     public Builder logInitializer(Runnable logInitializer) {
       this.logInitializer = checkNotNull(logInitializer, "logInitializer");
       return this;
diff --git a/elasticsearch/src/test/java/zipkin2/elasticsearch/ElasticsearchContainer.java b/elasticsearch/src/test/java/zipkin2/elasticsearch/ElasticsearchContainer.java
index 9514413..92c7941 100644
--- a/elasticsearch/src/test/java/zipkin2/elasticsearch/ElasticsearchContainer.java
+++ b/elasticsearch/src/test/java/zipkin2/elasticsearch/ElasticsearchContainer.java
@@ -46,7 +46,7 @@ class ElasticsearchContainer extends GenericContainer<ElasticsearchContainer> {
   static final Logger LOGGER = LoggerFactory.getLogger(ElasticsearchContainer.class);

   ElasticsearchContainer(int majorVersion) {
-    super(parse("ghcr.io/openzipkin/zipkin-elasticsearch" + majorVersion + ":3.0.5"));
+    super(parse("openzipkin/zipkin-elasticsearch" + majorVersion + ":test"));
     addExposedPort(9200);
     waitStrategy = Wait.forHealthcheck();
     withLogConsumer(new Slf4jLogConsumer(LOGGER));
@@ -129,11 +129,7 @@ class ElasticsearchContainer extends GenericContainer<ElasticsearchContainer> {
       ElasticsearchDependenciesJob.builder()
         .hosts(hostPort())
         .day(day)
-        .conf(Map.of(
-          ES_NODES_DISCOVERY, "false",
-          ES_NODES_WAN_ONLY, "true",
-          ES_HTTP_TIMEOUT, "5m" // default is 1m, allow longer in CI
-        ))
+        .conf(Map.of(ES_NODES_DISCOVERY, "false", ES_NODES_WAN_ONLY, "true"))
         .build().run();
     }
   }

running ITElasticsearchDependenciesHeavyV7 is fine in my IDE, but ITElasticsearchDependenciesHeavyV8 crashes, and in the docker logs you see this:

[2024-01-24T12:16:26,321][WARN ][r.suppressed             ] [zipkin-elasticsearch] path: /_template/zipkin-span_template, params: {name=zipkin-span_template}, status: 503
org.elasticsearch.discovery.MasterNotDiscoveredException: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:317) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:355) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:293) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:645) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[elasticsearch-8.12.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction.checkBlock(TransportGetIndexTemplatesAction.java:55) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction.checkBlock(TransportGetIndexTemplatesAction.java:30) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.checkBlockIfStateRecovered(TransportMasterNodeAction.java:136) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:201) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:173) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1157) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.getTemplates(AbstractClient.java:1502) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.action.admin.indices.RestGetIndexTemplateAction.lambda$prepareRequest$1(RestGetIndexTemplateAction.java:66) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94) ~[?:?]
    at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:73) ~[?:?]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
    at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]

Solving this way may lead to a solution.. note I tried various versions of ES 8.x with our docker image to no avail, I didn't dig into upstream etc.

codefromthecrypt commented 7 months ago

UPDATE: I think adding armeria client logging may help solve why this crashes on ES 7 but not 8. This is crashing during our index template initialization, and maybe it is something weird transport wise? I might be wrong, but anyway this is as far as I can go tonight.

with the following config (per latest commit), and using docker images built locally..

        .conf(Map.of(
          ES_NODES_DISCOVERY, "false",
          ES_NODES_WAN_ONLY, "true",
          ES_HTTP_TIMEOUT, "5m" // default is 1m, allow longer in CI
        ))

ITElasticsearchDependenciesHeavyV7 passes fine with the following docker logs

[2024-01-24T12:24:44,047][WARN ][o.e.d.r.RestController   ] [zipkin-elasticsearch] Legacy index templates are deprecated in favor of composable templates.
[2024-01-24T12:24:44,056][INFO ][o.e.c.m.MetadataIndexTemplateService] [zipkin-elasticsearch] adding template [zipkin-span_template] for index patterns [zipkin-span-*]
[2024-01-24T12:24:44,074][WARN ][o.e.d.r.RestController   ] [zipkin-elasticsearch] Legacy index templates are deprecated in favor of composable templates.
[2024-01-24T12:24:44,076][INFO ][o.e.c.m.MetadataIndexTemplateService] [zipkin-elasticsearch] adding template [zipkin-dependency_template] for index patterns [zipkin-dependency-*]
[2024-01-24T12:24:44,091][WARN ][o.e.d.r.RestController   ] [zipkin-elasticsearch] Legacy index templates are deprecated in favor of composable templates.
[2024-01-24T12:24:44,094][INFO ][o.e.c.m.MetadataIndexTemplateService] [zipkin-elasticsearch] adding template [zipkin-autocomplete_template] for index patterns [zipkin-autocomplete-*]
[2024-01-24T12:24:44,154][INFO ][o.e.c.m.MetadataCreateIndexService] [zipkin-elasticsearch] [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001] creating index, cause [initialize_data_stream], templates [.deprecation-indexing-template], shards [1]/[1]
[2024-01-24T12:24:44,160][INFO ][o.e.c.r.a.AllocationService] [zipkin-elasticsearch] updating number_of_replicas to [0] for indices [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001]
[2024-01-24T12:24:44,165][INFO ][o.e.c.m.MetadataCreateDataStreamService] [zipkin-elasticsearch] adding data stream [.logs-deprecation.elasticsearch-default] with write index [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001], backing indices [], and aliases []
[2024-01-24T12:24:44,234][INFO ][o.e.x.i.IndexLifecycleTransition] [zipkin-elasticsearch] moving index [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.deprecation-indexing-ilm-policy]
[2024-01-24T12:24:44,260][INFO ][o.e.x.i.IndexLifecycleTransition] [zipkin-elasticsearch] moving index [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [.deprecation-indexing-ilm-policy]
[2024-01-24T12:24:44,281][INFO ][o.e.x.i.IndexLifecycleTransition] [zipkin-elasticsearch] moving index [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.deprecation-indexing-ilm-policy]
[2024-01-24T12:24:44,359][INFO ][o.e.c.m.MetadataCreateIndexService] [zipkin-elasticsearch] [zipkin-span-2024-01-24] creating index, cause [auto(bulk api)], templates [zipkin-span_template], shards [5]/[1]
[2024-01-24T12:24:44,488][INFO ][o.e.c.m.MetadataMappingService] [zipkin-elasticsearch] [.ds-.logs-deprecation.elasticsearch-default-2024.01.24-000001/ftO8LkIsT_CufB7-3Yqpsw] update_mapping [_doc]
[2024-01-24T12:24:44,503][INFO ][o.e.c.m.MetadataMappingService] [zipkin-elasticsearch] [zipkin-span-2024-01-24/TB85HAJVQ4ecDiFWjiOybw] update_mapping [_doc]
[2024-01-24T12:24:44,506][INFO ][o.e.c.m.MetadataMappingService] [zipkin-elasticsearch] [zipkin-span-2024-01-24/TB85HAJVQ4ecDiFWjiOybw] update_mapping [_doc]
[2024-01-24T12:24:49,267][INFO ][o.e.c.m.MetadataCreateIndexService] [zipkin-elasticsearch] [.ds-ilm-history-5-2024.01.24-000001] creating index, cause [initialize_data_stream], templates [ilm-history], shards [1]/[0]
[2024-01-24T12:24:49,269][INFO ][o.e.c.m.MetadataCreateDataStreamService] [zipkin-elasticsearch] adding data stream [ilm-history-5] with write index [.ds-ilm-history-5-2024.01.24-000001], backing indices [], and aliases []
[2024-01-24T12:24:49,288][INFO ][o.e.x.i.IndexLifecycleTransition] [zipkin-elasticsearch] moving index [.ds-ilm-history-5-2024.01.24-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ilm-history-ilm-policy]
[2024-01-24T12:24:52,610][INFO ][o.e.c.m.MetadataCreateIndexService] [zipkin-elasticsearch] [zipkin-dependency-2024-01-24] creating index, cause [api], templates [zipkin-dependency_template], shards [5]/[1]
[2024-01-24T12:24:52,734][INFO ][o.e.x.i.IndexLifecycleTransition] [zipkin-elasticsearch] moving index [.ds-ilm-history-5-2024.01.24-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [ilm-history-ilm-policy]
[2024-01-24T12:24:52,796][INFO ][o.e.x.i.IndexLifecycleTransition] [zipkin-elasticsearch] moving index [.ds-ilm-history-5-2024.01.24-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [ilm-history-ilm-policy]
[2024-01-24T12:24:53,014][INFO ][o.e.c.m.MetadataDeleteIndexService] [zipkin-elasticsearch] [zipkin-span-2024-01-24/TB85HAJVQ4ecDiFWjiOybw] deleting index
[2024-01-24T12:24:53,049][INFO ][o.e.c.m.MetadataDeleteIndexService] [zipkin-elasticsearch] [zipkin-dependency-2024-01-24/UJXz0spdTqaaJkyFomBYXQ] deleting index

but ITElasticsearchDependenciesHeavyV8 crashes with client side log:

com.linecorp.armeria.client.ResponseTimeoutException
    at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:38)
    at com.linecorp.armeria.internal.common.DefaultCancellationScheduler.invokeTask(DefaultCancellationScheduler.java:495)
    at com.linecorp.armeria.internal.common.DefaultCancellationScheduler.lambda$init0$2(DefaultCancellationScheduler.java:129)
    at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:300)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: org.opentest4j.TestAbortedException: Could not connect to storage, skipping test: null
        at zipkin2.storage.ITStorage.checkStorage(ITStorage.java:66)
        at zipkin2.storage.ITStorage.doInitializeStorage(ITStorage.java:59)
        at zipkin2.storage.ITStorage.initializeStorageForTest(ITStorage.java:48)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    Caused by: com.linecorp.armeria.client.ResponseTimeoutException
        at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:38)
        at com.linecorp.armeria.internal.common.DefaultCancellationScheduler.invokeTask(DefaultCancellationScheduler.java:495)
        at com.linecorp.armeria.internal.common.DefaultCancellationScheduler.lambda$init0$2(DefaultCancellationScheduler.java:129)
        at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45)

and docker (server side log) as below.

[2024-01-24T12:25:36,548][INFO ][o.e.n.Node               ] [zipkin-elasticsearch] started {zipkin-elasticsearch}{SJSvIoCYScKvUHkXeQw7OA}{7iv9frw-Q-aueoaLcHj-ng}{zipkin-elasticsearch}{172.17.0.3}{172.17.0.3:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}{transform.config_version=10.0.0, xpack.installed=true}
[2024-01-24T12:26:09,545][WARN ][r.suppressed             ] [zipkin-elasticsearch] path: /_template/zipkin-span_template, params: {name=zipkin-span_template}, status: 503
org.elasticsearch.discovery.MasterNotDiscoveredException: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:317) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:355) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:293) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:645) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[elasticsearch-8.12.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction.checkBlock(TransportGetIndexTemplatesAction.java:55) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction.checkBlock(TransportGetIndexTemplatesAction.java:30) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.checkBlockIfStateRecovered(TransportMasterNodeAction.java:136) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:201) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:173) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1157) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.getTemplates(AbstractClient.java:1502) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.action.admin.indices.RestGetIndexTemplateAction.lambda$prepareRequest$1(RestGetIndexTemplateAction.java:66) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94) ~[?:?]
    at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:73) ~[?:?]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431) ~[elasticsearch-8.12.0.jar:?]
    at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
    at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
reta commented 7 months ago

I would not be opposed to changing this out to opensearch or something else either, but anyway fresh eyes might find a way forward I didn't.

Sure @codefromthecrypt , I will take a look shortly

reta commented 7 months ago

@codefromthecrypt I think https://github.com/openzipkin/zipkin/pull/3708/files#r1465403498 should solve the issue

codefromthecrypt commented 7 months ago

ok thanks to @reta the upstream test image works fine now. I left the jupiter and logging changes here and updated the description.