search

JPro-one / JPro-Tickets

The right place to report about bugs or suggest improvements for JPro.
https://www.jpro.one
9 stars 4 forks source link

JPro server unresponsive/crash #64

Open Hugolarson opened 4 years ago

Hugolarson commented 4 years ago

I have a pretty complex application and sometimes it just enters a state where it freezes. JPro process is still running without a hint of problem in logs but the application will not become reachable until restart of JPro server.

I think I have seen this only on Linux/Nginx. Trying to reach it gives "502 bad gateway".

This is Nginx log 2020/07/17 04:26:27 [error] 653#653: *3190615 connect() failed (111: Connection refused) while connecting to upstream, client: 124.41.4.4, server: jproapp.domain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "jproapp.domain.com".

What can we do triangulate the problem as there is no hint in logs?

FlorianKirmaier commented 4 years ago

There are some questions you can ask. Is the JProServer still available? /status/alive Is the JavaFXThread stuck? /info/fxstack How is memory usage? /status check the heap dump for memory leaks: /info/heapdumps/heapdump.hprof Is the process of the RUNNING.pid still running? Check out our documentation page: https://www.jpro.one/?page=docs/current/2.5/DEBUGGING_AND_TESTING

In the form of paid consulting, we could probably quickly help you fix your problem, but from the ticket-system i can only help you by asking some basic questions.

Hugolarson commented 4 years ago

It still running. Tried to start it but it's still unreachable so it's running. Cannot access it to check status etc. Gives Nginx "502 bad gateway"

Can I check these things on file system?

This time I found this exception in log. 2020-07-17 12:15:38,401 [ERROR] from com.jpro.meta.tk.SceneInputListener in JavaFX Application Thread - Error while processing the input: MouseEventToken(Mou seEntered,25.600000381469727,701.2000122070312,MouseButtonToken(2),false,false,false,false,false,false,false,false) app: stage title: Debiteringshistorik java.lang.NullPointerException: null at simplefx.util.ReflectionUtil$UseReflection.applyDynamic(ReflectionUtil.scala:18) at com.jpro.meta.tk.UtilSceneTKListener$ExtendView.view(UtilSceneTKListener.scala:24) at com.jpro.meta.tk.UtilSceneTKListener$ExtendView.isMonocle(UtilSceneTKListener.scala:25) at com.jpro.meta.tk.SceneInputListener.processInputInScene(SceneInputListener.scala:127) at com.jpro.meta.tk.SceneInputListener.processInput(SceneInputListener.scala:117) at com.jpro.meta.tk.SceneInputListener.processInputs(SceneInputListener.scala:98) at com.jpro.meta.tk.SceneInputListener.$anonfun$new$4(SceneInputListener.scala:39) at simplefx.cores.standard.parts.CoreTimeEvent.$anonfun$runAsAnimation$1(CoreTimeEvent.scala:64) at simplefx.cores.standard.parts.CoreTimeEvent$$anon$2.handle(CoreTimeEvent.scala:46) at javafx.animation.AnimationTimer$AnimationTimerReceiver.lambda$handle$0(AnimationTimer.java:57) at java.security.AccessController.doPrivileged(Native Method) at javafx.animation.AnimationTimer$AnimationTimerReceiver.handle(AnimationTimer.java:56) at com.sun.scenario.animation.AbstractMasterTimer.timePulseImpl(AbstractMasterTimer.java:357) at com.sun.scenario.animation.AbstractMasterTimer$MainLoop.run(AbstractMasterTimer.java:267) at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:557) at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:541) at com.sun.javafx.tk.quantum.QuantumToolkit.pulseFromQueue(QuantumToolkit.java:534) at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$runToolkit$11(QuantumToolkit.java:340) at com.sun.glass.ui.monocle.RunnableProcessor.runLoop(RunnableProcessor.java:92) at com.sun.glass.ui.monocle.RunnableProcessor.run(RunnableProcessor.java:51) at java.lang.Thread.run(Thread.java:748)

FlorianKirmaier commented 4 years ago

Can you access these debug-pages before it's crashing? Maybe you will notice something suspicious before it's happening. You should also be able to check the memory-usage with some command-line tools and check how close it is to the maximum.

Here are some shortcuts:

jstack `cat RUNNING_PID`
jcmd  `cat RUNNING_PID` GC.heap_info
jmap -dump:format=b,file=heapdump.hprof `cat RUNNING_PID`

I will add them to our debug-documentation because they are often useful.