restheart has no response after running a while

[darlingpretty168]

Our web application calls restheart to grab data from Mongo. However, after restheart runs or idle couple hours, it stop responding the client request somehow. Any advice? Thanks.

[mkjsix]

Is there any firewall / proxy in between your client and the server?

[darlingpretty168]

No...

[ujibang]

Please provide more information: restheart version, OS, java version, hw configuration (cpu, memory), network configuration, restheart logs, system logs, running on-premises or in a cloud, cloud type, anything that could help...

Also what response does the client get? Can you try invoking restheart with curl or httpie?

We manage several restheart instances and we never experienced this issue. Maybe a network issue??

[darlingpretty168]

version 1.0.3 and server is Ubuntu. The log is as below.

07:04:32.116 [main] INFO org.restheart.Bootstrapper - URL / bound to MongoDB resource * 07:04:32.270 [main] INFO org.restheart.Bootstrapper - Embedded static resources browser extracted in /tmp/restheart-6935094770078716802 07:04:32.292 [main] INFO org.restheart.Bootstrapper - URL /browser bound to static resources browser. access manager: false 07:04:32.295 [main] INFO org.restheart.Bootstrapper - URL /_logic/ping bound to application logic handler org.restheart.handlers.applicationlogic.PingHandler. access manager: false 07:04:32.302 [main] INFO org.restheart.Bootstrapper - URL /_logic/roles bound to application logic handler org.restheart.handlers.applicationlogic.GetRoleHandler. access manager: false 07:04:32.303 [main] INFO org.restheart.Bootstrapper - URL /_logic/ic bound to application logic handler org.restheart.handlers.applicationlogic.CacheInvalidator. access manager: true 07:04:32.481 [main] INFO org.restheart.Bootstrapper - RESTHeart started ** 14:39:29.007 [Thread-2] INFO org.restheart.Bootstrapper - Stopping RESTHeart... 14:39:29.008 [Thread-2] INFO org.restheart.Bootstrapper - Waiting for pending request to complete (up to 1 minute)... 14:40:29.008 [Thread-2] INFO org.restheart.Bootstrapper - Flushing and closing the MongoDB client... 14:40:29.012 [Thread-2] INFO org.restheart.Bootstrapper - Removing the pid file /tmp/restheart--588166521.pid 14:40:29.012 [Thread-2] INFO org.restheart.Bootstrapper - Cleaning up temporary directories... 14:40:29.015 [Thread-2] INFO org.restheart.Bootstrapper - RESTHeart stopped *****

[ujibang]

Can't reproduce the problem.

Closing for now, if @darlingpretty168 has more info, can reopen it...

[jnmayur]

Hi I am facing a similar issue. Can someone please see how can this be looked into further. OS: Unix RestHeart Version: 3.4.2 Run manually and not in docker.

08:57:06.351 [XNIO-1 task-7] ^[[34mINFO ^[[0;39m o.r.handlers.RequestLoggerHandler - GET http://ec2-54-87-34-182.compute-1.amazonaws.com/ from /60.191.38.77:49572 => status=^[[31;1m401^[[m elapsed=0ms contentLength=0 09:15:51.326 [XNIO-1 task-3] ^[[34mINFO ^[[0;39m o.r.handlers.RequestLoggerHandler - GET http://54.87.34.182:8080/ from /187.74.99.215:41834 => status=^[[31;1m401^[[m elapsed=0ms contentLength=0 10:03:22.198 [XNIO-1 task-1] ^[[34mINFO ^[[0;39m o.r.handlers.RequestLoggerHandler - GET http://54.87.34.182:8080/ from /84.55.69.65:43845 => status=^[[31;1m401^[[m elapsed=0ms contentLength=0 10:27:50.178 [Thread-1] ^[[34mINFO ^[[0;39m org.restheart.Bootstrapper - Stopping RESTHeart... 10:27:50.178 [Thread-1] ^[[34mINFO ^[[0;39m org.restheart.Bootstrapper - Waiting for pending request to complete (up to 1 minute)... 10:27:50.178 [Thread-1] ^[[34mINFO ^[[0;39m org.restheart.Bootstrapper - Closing MongoDB client connections... 10:27:50.182 [Thread-1] ^[[34mINFO ^[[0;39m org.restheart.Bootstrapper - Removing the pid file /tmp/restheart--496984543.pid 10:27:50.182 [Thread-1] ^[[34mINFO ^[[0;39m org.restheart.Bootstrapper - Cleaning up temporary directories... 10:27:50.197 [Thread-1] ^[[34mINFO ^[[0;39m org.restheart.Bootstrapper - ^[[32;1mRESTHeart stopped^[[m

[ujibang]

Hi @jnmayur,

from your log it seems that at 10:27:50 the RESTHeart process has been stopped, for instance it received a SIGKILL signal.

How did you start RESTHeart? Was it from a terminal? Maybe the terminal was closed. In this case I suggest you to start it with the bin/start.sh script or use nohup

I personally have systems with RESTHeart running from more than 2 months, so I suggest you to check how you run it.

[jnmayur]

Hi @ujibang

This is how I had started RestHeart nohup java -server -jar /home/sirion/restheart/restheart.jar /home/sirion/restheart/etc/restheart.yml &

Also, I found the logs of MongoDB becuase of which it got the signal to stop. 2019-02-04T10:23:16.604+0000 I NETWORK [conn663] end connection 169.254.44.150:41040 (43 connections now open) 2019-02-04T10:25:15.795+0000 I NETWORK [listener] connection accepted from 169.254.44.150:44040 #664 (44 connections now open) 2019-02-04T10:25:15.836+0000 I NETWORK [conn664] received client metadata from 169.254.44.150:44040 conn: { driver: { name: "mongo-java-driver", version: "3.6.4" }

, os: { type: "Linux", name: "Linux", architecture: "amd64", versio n: "4.15.0-43-generic" }

, platform: "Java/Oracle Corporation/9.0.4+11" } 2019-02-04T10:27:49.817+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends 2019-02-04T10:27:49.817+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... 2019-02-04T10:27:49.817+0000 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock 2019-02-04T10:27:49.817+0000 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture 2019-02-04T10:27:49.819+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down 2019-02-04T10:27:50.022+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock... 2019-02-04T10:27:50.022+0000 I CONTROL [signalProcessingThread] now exiting 2019-02-04T10:27:50.022+0000 I CONTROL [signalProcessingThread] shutting down with code:0 2019-02-04T10:28:43.781+0000 I CONTROL [main] SERVER RESTARTED 2019-02-04T10:28:44.090+0000 I CONTROL [initandlisten] MongoDB starting : pid=1007 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-60-62-125 2019-02-04T10:28:44.090+0000 I CONTROL [initandlisten] db version v3.6.3 2019-02-04T10:28:44.090+0000 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5 2019-02-04T10:28:44.090+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014 2019-02-04T10:28:44.090+0000 I CONTROL [initandlisten] allocator: tcmalloc 2019-02-04T10:28:44.090+0000 I CONTROL [initandlisten] modules: none

Not sure, how to restart RestHeart on auto restart of Mongo.

[ujibang]

maybe you can run RESTHeart with systemd

check the etc/daemon folder...

[mkjsix]

By the way, from your logs it seems you are running RESTHeart with Java 9 but note that 3.4.2 has only been tested with Java 8

"Java/Oracle Corporation/9.0.4+11"

It could work anyway, but I'm not 100% sure. Version 3.7 is tested also with OpenJDK 11.