ES Nodes won't start on 8.11.0 BC1 in cloud

[bhavyarm]

Description

ES nodes are not starting in cloud for BC1 of 8.11.0

This is the fatal error:

[instance-0000000001] fatal error in thread [elasticsearch[instance-0000000001][search][T#3]], exiting
java.lang.ExceptionInInitializerError: null
    at co.elastic.apm.agent.profiler.ProfilingActivationListener.beforeActivate(ProfilingActivationListener.java:44) ~[?:?]
    at co.elastic.apm.agent.impl.ActiveStack.triggerActivationListeners(ActiveStack.java:139) ~[?:?]
    at co.elastic.apm.agent.impl.ActiveStack.activate(ActiveStack.java:101) ~[?:?]
    at co.elastic.apm.agent.impl.ElasticApmTracer.activate(ElasticApmTracer.java:867) ~[?:?]
    at co.elastic.apm.agent.opentelemetry.context.OTelContextStorage.attach(OTelContextStorage.java:62) ~[?:?]
    at io.opentelemetry.context.Context.makeCurrent(Context.java:202) ~[?:?]
    at org.elasticsearch.telemetry.apm.internal.tracing.APMTracer.withScope(APMTracer.java:286) ~[?:?]
    at org.elasticsearch.telemetry.tracing.Tracer.withScope(Tracer.java:184) ~[elasticsearch-8.11.0.jar:?]

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[stu-elastic]

More details in the error

[2023-10-06T14:24:53,633][ERROR][org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler] [instance-0000000001] fatal error in thread [elasticsearch[instance-0000000001][system_read][T#1]], exiting
java.lang.ExceptionInInitializerError: null
    at co.elastic.apm.agent.profiler.ProfilingActivationListener.beforeActivate(ProfilingActivationListener.java:44) ~[?:?]
    at co.elastic.apm.agent.impl.ActiveStack.triggerActivationListeners(ActiveStack.java:139) ~[?:?]
    at co.elastic.apm.agent.impl.ActiveStack.activate(ActiveStack.java:101) ~[?:?]
    at co.elastic.apm.agent.impl.ElasticApmTracer.activate(ElasticApmTracer.java:867) ~[?:?]
    at co.elastic.apm.agent.opentelemetry.context.OTelContextStorage.attach(OTelContextStorage.java:62) ~[?:?]
    at io.opentelemetry.context.Context.makeCurrent(Context.java:202) ~[?:?]
    at org.elasticsearch.telemetry.apm.internal.tracing.APMTracer.withScope(APMTracer.java:286) ~[?:?]
    at org.elasticsearch.telemetry.tracing.Tracer.withScope(Tracer.java:184) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:662) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:542) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:51) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:48) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.0.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.UnsupportedOperationException: Cannot define class using reflection: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
    at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Unavailable.defineClass(ClassInjector.java:1472) ~[?:?]
    at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw(ClassInjector.java:284) ~[?:?]
    at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject(ClassInjector.java:118) ~[?:?]
    at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load(ClassLoadingStrategy.java:241) ~[?:?]
    at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load(ClassLoadingStrategy.java:148) ~[?:?]
    at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:101) ~[?:?]
    at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:6325) ~[?:?]
    at co.elastic.apm.agent.sdk.internal.ThreadUtil.generateVirtualChecker(ThreadUtil.java:65) ~[?:?]
    at co.elastic.apm.agent.sdk.internal.ThreadUtil.<clinit>(ThreadUtil.java:35) ~[?:?]
    ... 18 more

This is https://github.com/elastic/elasticsearch/issues/100072, fixed by https://github.com/elastic/elasticsearch/pull/100232

[stu-elastic]

This is fixed by the backport in #100419