Reproducible hang on Debian 9, oracle-java-13 and above

RIPE-NCC / rpki-validator-3

RIPE NCC RPKI Validator 3

Other

63 stars 27 forks source link

Reproducible hang on Debian 9, oracle-java-13 and above #277

Open geeohgeegeeoh opened 4 years ago

geeohgeegeeoh commented 4 years ago

For the current code, I can reproduce a hang on oracle-java-13, oracle-java-14 and oracle-java-15.

oracle-java-11 is fine.

On FreeBSD 12, more recent OpenJDK to equivalent levels are fine btw.

I can give you a jstack thread dump if you want.

lolepezy commented 4 years ago

I can give you a jstack thread dump if you want.

That would be very helpful! Especially for jdk 14.

geeohgeegeeoh commented 4 years ago

Here's a jthread trace java-threads.txt

ties commented 4 years ago

Hi,

Two quick questions:

On what kind of machine (distribution/number of CPU cores) do you encounter this issue?
How long after startup do you encounter this?

This will help us when attempting to reproduce it :)

geeohgeegeeoh commented 4 years ago

Its a 48 core Dell running Debian, with around 250GB of memory.

It happens within 5min. From clean start, attempting to bootstrap state. Sometimes it can pass through, but usually it gets stuck.

On Mon, Sep 28, 2020 at 11:18 PM Ties de Kock notifications@github.com wrote:

Hi,

Two quick questions:

On what kind of machine (distribution/number of CPU cores) do you encounter this issue?

How long after startup do you encounter this?

This will help us when attempting to reproduce it :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RIPE-NCC/rpki-validator-3/issues/277#issuecomment-700000895, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABORQ7AWZ3727GD4NWL3LRTSICEKRANCNFSM4RVD6AJQ .

ties commented 4 years ago

I can reproduce this on a machine with a high number of cores (R730xd/28 cores/56 hyperthreads/openjdk 14). It seems like it gets stuck with the high amount of threads created and this is a symptom of an underlying bug.

I could work around it by limiting the number of threads (by setting _JAVA_OPTIONS="-XX:ActiveProcessorCount=16" in an environment variable). Please let me know if that is a suitable workaround for you for now.

@erikrozendaal could you take a look?