Augmentation of quarkus app on Red Hat 8.5 stuck

esorio commented 1 year ago

Describe the bug

Hi, I have a project with the quarkus-kubernetes extension, if I try to perform an "augmentation" on RHEL 8.5 the process remains stuck (on ubuntu 20.04 it works as expected).

The augmentation seems completed successfully (I can read the following log) INFO [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 1659ms but the process won't exit.

Expected behavior

The augmentation process completes with exit code 0.

Actual behavior

The augmentation process remains stuck.

How to Reproduce?

Steps to reproduce the problem:

checkout the reproducer https://github.com/esorio/quarkus-augmentation-stuck
compile with the mutable jar option ./gradlew build -Dquarkus.package.type=mutable-jar
perform the augmentation java -jar -Dquarkus.launch.rebuild=true build/quarkus-app/quarkus-run.jar

Output of `uname -a` or `ver`

Linux rhel8_5 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of `java -version`

openjdk version "11.0.15" 2022-04-19

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.1.3

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

gradle 8.1.1

Additional information

No response

quarkus-bot[bot] commented 1 year ago

/cc @Sgitario (kubernetes), @geoand (kubernetes), @iocanel (kubernetes)

geoand commented 1 year ago

Is there any chance you can inspect the process and see what is stuck?

cc @maxandersen @tqvarnst

esorio commented 1 year ago

I can see the following list of threads attaching VisualVM to the running process, don't know if it helps.

threads

geoand commented 1 year ago

A thread dump (via jstack for example) might help

esorio commented 1 year ago

Hope it helps

threaddump.txt

geoand commented 1 year ago

Thanks for the information, unfortunately there isn't much info there...

@rsvoboda can anyone from your team check and see if you can reproduce the problem?

rsvoboda commented 1 year ago

Back from my vacation. I will put this on my list for this week.

@geoand fyi, there is an internal beaker service where you can provision certain version of RHEL for ~24h.

geoand commented 1 year ago

That's nice to know, thanks!

rsvoboda commented 1 year ago

I used the reproducer and couldn't hit the described issue on RHEL 8.5

[quarkus-augmentation-stuck]# java -jar -Dquarkus.launch.rebuild=true build/quarkus-app/quarkus-run.jar
Invalid AnsiLogger Stream -> Swapping to default sdt out logger.
[WARN] Could not detect project version. Using 'latest'.
2023-07-18 08:43:37,409 WARN  [io.qua.config] (build-13) Unrecognized configuration key "quarkus.launch.rebuild" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2023-07-18 08:43:38,867 INFO  [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 3142ms
[quarkus-augmentation-stuck]#

[quarkus-augmentation-stuck]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 (Ootpa)
[quarkus-augmentation-stuck]# java -version
openjdk version "17" 2021-09-14
OpenJDK Runtime Environment 21.9 (build 17+35)
OpenJDK 64-Bit Server VM 21.9 (build 17+35, mixed mode, sharing)

esorio commented 1 year ago

Hi, thanks for the test.

I tried to reproduce the problem in a clean environment...

I downloaded the RHEL 8.5 iso from http://calipso.linux.it.umich.edu/pulp/isos/UM/Library/content/dist/rhel8/8/x86_64/baseos/iso/rhel-8.5-x86_64-dvd.iso.

I installed Java 17

openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7)
OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)

I compiled the reproducer and running the augmentation command the problem is still the same

I tried both locally on virtualbox 6.1 / 7 and on VMware ESXi 7.

It worth to note that if I remove from the build.gradle the implementation 'io.quarkus:quarkus-kubernetes' dependency the problem does not occur.

gsmet commented 1 year ago

One thing that could help is if you could do a kill -3 <stuck process> and post the thread dump.

esorio commented 1 year ago

This is the thread dump after killing the stuck process.

thread_dump_kill3.txt

quarkusio / quarkus