Native Build in OpenShift is failing

[aklemp]

Describe the bug Building a native Quarkus application in OpenShift 3.11 is failing. The same works locally without problems (on Debian with Docker build and GraalVM 19.3.1-java8).

Expected behavior The build is successful like it is locally.

Actual behavior There are a lot of errors but it seems like OpenShift is trying to augment the application with features, that are not supported natively.

[getting-started-1.0-SNAPSHOT-runner:205]     analysis: 128,603.97 ms
Error: Unsupported features in 5 methods
Detailed message:
Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time.  To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-build-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace:  object java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
    object java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
    object java.util.concurrent.LinkedBlockingQueue
    object com.instana.agent.loader.v1_3_23.AgentUrlLogger
    object com.instana.agent.loader.v1_3_23.AgentLogDispatcher
    object java.util.InstanaLoggerV4
    field java.util.InstanaLoggerV4.INSTANCE
Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time.  To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-build-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace:  object sun.net.www.http.KeepAliveCache
    field sun.net.www.http.HttpClient.kac
Error: No instances of java.net.SocketInputStream are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use -H:+TraceClassInitialization.
Trace:  object java.io.BufferedInputStream
    object sun.net.www.http.HttpClient
    object sun.net.www.http.KeepAliveEntry
    object java.lang.Object[]
    object sun.net.www.http.ClientVector
    object java.util.HashMap$Node
    object java.util.HashMap$Node[]
    object sun.net.www.http.KeepAliveCache
    field sun.net.www.http.HttpClient.kac
Error: No instances of java.net.SocketOutputStream are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use -H:+TraceClassInitialization.
Trace:  object java.io.BufferedOutputStream
    object java.io.PrintStream
    object sun.net.www.http.HttpClient
    object sun.net.www.http.KeepAliveEntry
    object java.lang.Object[]
    object sun.net.www.http.ClientVector
    object java.util.HashMap$Node
    object java.util.HashMap$Node[]
    object sun.net.www.http.KeepAliveCache
    field sun.net.www.http.HttpClient.kac
Error: No instances of java.net.SocksSocketImpl are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use -H:+TraceClassInitialization.
Trace:  object java.net.Socket
    object sun.net.www.http.HttpClient
    object sun.net.www.http.KeepAliveEntry
    object java.lang.Object[]
    object sun.net.www.http.ClientVector
    object java.util.HashMap$Node
    object java.util.HashMap$Node[]
    object sun.net.www.http.KeepAliveCache
    field sun.net.www.http.HttpClient.kac

To Reproduce

• Create new project with the contents of https://github.com/quarkusio/quarkus-quickstarts/tree/master/getting-started.
• Have to set MAVEN_MIRROR_URL and HTTP_PROXY_NONPROXYHOST in .s2i/environment in our environment.
• Create a pipeline (we are using GitLab CI with necessary plugins to interact with openshift). The relevant commands in the pipeline are.

  oc create imagestreamtag ubi-quarkus-native-s2i:19.3.1-java8 --from-image=quay.io/quarkus/ubi-quarkus-native-s2i:19.3.1-java8
  oc process -f openshift/build-native.yaml | oc apply -f -
  oc start-build getting-started --from-dir=. --wait

  Build-Config in openshift/build-native.yaml:

  
  ---

kind: Template apiVersion: v1 metadata: name: getting-started labels: template: getting-started

objects:

• kind: ImageStream apiVersion: v1 metadata: name: getting-started labels: app: getting-started spec: lookupPolicy: local: true dockerImageRepository: '' tags:
  • name: latest
• kind: BuildConfig apiVersion: v1 metadata: name: getting-started labels: app: getting-started spec: runPolicy: SerialLatestOnly successfulBuildsHistoryLimit: 2 failedBuildsHistoryLimit: 1 source: binary: {} type: Binary strategy: sourceStrategy: from: kind: ImageStreamTag name: ubi-quarkus-native-s2i:19.3.1-java8 type: Source output: to: kind: ImageStream name: getting-started resources: requests: memory: 2Gi limits: memory: 10Gi

Configuration Using a two-stage build to create a minimal image. The problem here is already building the native application (the first stage).

Screenshots n/a

Environment (please complete the following information):

• Quarkus 1.2.0.Final
• GraalVM 19.3.1-java8
• OpenShift 3.11
• builder image ubi-quarkus-native-s2i:19.3.1-java8

Additional context n/a

[cescoffier]

I just tried (using minishift) and it was working for me. I'm surprised by com.instana.agent.loader, are you sure you don't have an agent or extra dependency in your application?

[aklemp]

The project code is exactly as described but I suspect some instrumentation magic in OpenShift (because Instana is deployed somewhere). In the meantime I found this article. I have to try around a bit and see if this issue can be resolved.

[dzou]

Did you ever figure this issue out? I happened to stumble on the same issue.

[cody-pope]

@aklemp @dzou I'm running into the exact same issue, and it only happens when the Instana agent is enabled. Were you guys able to resolve it?

[aklemp]

Unfortunately, I didn't find a solution yet. Played around with some of the configuration settings but nothing seemed to help.

[dzou]

I'll try my best to help - basically I had the same situation where the build for my application worked locally but failed in the CI. I didn't get this error in the context of OpenShift though.

For me the problem was I had a class in my project which should have been initialized at runtime instead of build-time. See: https://www.graalvm.org/reference-manual/native-image/ClassInitialization/

The runtime vs. build-time initialization concept is important to understand to fixing the problem. Basically Quarkus defaults to initializing everything at build-time, which means that all static initializers and variables in a class are initialized when the native image is built. (Like many times, static variables are often constants that evaluate to the same value everytime the code is run, so why not initialize it once at build-time, then cache that value in the image so that in the future you don't have to redo this work.) This is not what the standard JVM does - which is to initialize a class (run its static blocks and initialize static variables) the first time it is referenced in the program's execution at runtime.

However, clearly many static variables must be initialized at runtime. Things like timestamps, or values whose initialization depend on a socket connection (such as in this case).

====

For my problem, I don't understand why it worked locally for me but not in the CI build.

I basically had a situation like this:

class Blah {

private static XClient client = initializeClient();

}

But the initializeClient() call makes some remote calls to some external services and must be initialized at runtime instead of build-time. It is what created those "socket" objects that the error refers to.

So I simply added --initialize-at-run-time=my.package.Blah to the graalvm compiler args and it worked.

It is often challenging to debug things like this. One thing you can try is to set the --debug-attach=5005 setting on the compiler and then set a breakpoint at sun.net.www.http.HttpClient.kac, run the native image build, and then see which class in the call stack triggered the initialization of the HttpClient.

But I think the solution is you just need to add another class as an argument to --initialize-at-run-time; It's likely this class is something in your own code base and not one of your dependencies.

[andr3adc]

Hi, I reply to this old thread because is the only one on the web reporting this issue. I solved the issue disabling the Instana agent only on the native-image building processes as following.

On the Intana agent configuration.yml:

com.instana.ignore:
  processes:
    - 'native-image'
    - '/opt/graalvm-ce-java11-22.2.0/bin/java'

(Change the path with your GraalVM intallation folder)

Probably the original issue is already solved, but I hope it helps someone :)

[geoand]

Is this still an issue?

[aklemp]

I'm not in that project anymore, so I cannot tell, if the problem is resolved...

[geoand]

Got it, thanks for the update.