OutOfMemory when after (19 to 25h) because process/resource limits reached

[StefanSchubert]

Hi there,

first thanks a lot for offering a java interface to PIs GPIO - this is really great and I'm quite happy with it 👍🏻

However from time to time my pi crashes and it seems that there is a bug or miss configuration in the event handling of pi4j.

Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: [80584,872s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: Exception in thread "pi4j-gpio-event-executor-0" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.lang.Thread.start0(Native Method)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.lang.Thread.start(Thread.java:803)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.security.AccessController.doPrivileged(Native Method)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: #011at java.base/java.lang.Thread.run(Thread.java:834)
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: [80584,882s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Mar  6 12:42:53 MediaParkPiBell pibell-service.jar[1521]: Exception in thread "pi4j-gpio-event-executor-1" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

Has someone observed this behavior before? I'm using the core lib in version 1.4 on a pi modell 3. My Application is just reading out a single GPIO Pin which works quite well. In HTOP on the PI I can observer a more or less steady number of threads belonging to the app. When the event I listen for occurs I see plus 2-3 more threads but they will disappear after a couple of minutes. I would say the number of threads looks steady. However after running round about a day the pay crashes and I can observe the stacktrace in syslog.

Ah and I added actuator/prometheus/grafana to observer if there is a problem with heap sizes etc...everything there is within bounds. No much memory consumption, nice GC runs etc.

Greeting from Germany Stefan

[StefanSchubert]

Update:

I did some analysis in between and found a solution to the issue, which has nothing do do with the pi4j lib so the issue is closed. For those who are curious:

My SpringBoot App using the pi4j runs on openJDK 11 which causes that Apps jar file is in fact being extracted at /tmp as well as hsperf Data along with the tomcat folders to store tmp upload files. However the App is accessing the exploded Jar libs and the JVM hsperf at runtime. Now there are default cronjobs of the debian on the pi which cleans up /tmp data. Now guess what? Sure the App crashed and it seems that it do it in such a way that will slow down the pi as well through "overheating" so that is is no longer responding. I fixed the issue in my case by disabing hsperf and moving the exploded jars by making use of -Djava.io.tempdir

Took me quite a while to figure that out %-)

[FDelporte]

Thanks for the update @StefanSchubert and clear description! I wrote a proof-of-concept Spring application, so will definitely take a look into this "-Djava.io.tempdir" argument.

[StefanSchubert]

Welcome @FDelporte , here are the exact JVM config: JAVA_OPTS="-Djava.io.tmpdir=/var/pibell/tmp -XX:-UsePerfData"

one problem might be, that now through restarts of the app, the new tmp dir will grow with exploded libs. You may consider own housekeeping for that. Take also into account that the behavior according exploding the JAR seems to differ between JVMs. I observed it with openjdk 11.0.9.1 2020-11-04 and I haven't quite understood what leads to the exploding jar behavior. I have severals Apps running on pis all based on spring boot and though the poms are almost identical, all but this one app, leave the jar (which is built as executable to run as a service) untouched.