search

hobuinc / greyhound

Greyhound is a point cloud streaming server. It should be considered deprecated for now. Use Entwine and Entwine Point Tile directly if you just want to serve point cloud web services.
https://greyhound.io
Apache License 2.0
131 stars 51 forks source link

Greyhound stops responding #55

Closed Hisol closed 6 years ago

Hisol commented 6 years ago

After upgrading Greyhound to be able to use the nativeBounds in queries we are now seeing issues where Greyhound will work for a period of time and then just stop responding. The docker container is still there and running but Greyhound isn't doing anything. Checking the logs of the container after it has been stopped and there is no error message.

If I kill the container and restart it, all is fine again for a period of time and then it stops. This goes for PoTree as well as my own application.

To test for it being alive I have tried issuing an info command on a resource (I assume these are quite light weight? unless there is a server status type request I could make?) and the TCP-IP connection establishes but I get nothing back.

I don't have a timescale for this happening alas, it's quite intermittent that Greyhound is called on and so far we have not needed to monitor it for responsiveness.

connormanning commented 6 years ago

I saw this a couple of times as well, and added some extra logging (as well as some unrelated enhancements) and haven't seen it since. Can you try updating to the very latest Greyhound image which I've just pushed, and if you see this failure again, paste any error-like messages from the end of the server logs?

Hisol commented 6 years ago

OK, have grabbed latest and restarted. I doubt it will get much use over the weekend, but if I see any more lock ups etc I will paste errors etc here. Thanks

Hisol commented 6 years ago

Hi @connormanning Our Greyhound server stopped responding again after upgrading to the newer version.

I stopped the container with docker stop before starting up a fresh instance. Checking output with docker logs shows no error messages, just the last few requests that it responded to.

68x31/info: 1 ms 68x31/read: 3 ms D: [all, 8) P: 19 68x32/read: 1460 ms D: [all, 14) P: 87428 68x31/read: 3 ms D: [all, 10) P: 235 68x31/read: 9 ms D: [all, 12) P: 2779 68x31/info: 1 ms 68x31/read: 3 ms D: [all, 8) P: 16 68x31/read: 3 ms D: [all, 10) P: 220 68x31/read: 9 ms D: [all, 12) P: 2892

Hisol commented 6 years ago

I also just had the fresh restarted version die within 13 minutes of being started.. again, no error output just logs of last activity.

Hisol commented 6 years ago

Sorry to keep bringing this up, have you made any progress with this yet? At the moment Greyhound is staying up for anything between 20 seconds and several hours, I have not yet seen any form of error message output. Sometimes the problem occurs just after it has served data, some times it is when it is sweeping (from the output I can see).

Thanks

Hisol commented 6 years ago

Have not seen this issue since your help, many thanks.