monarch-initiative / monarch-app

Monarch Initiative website and API
https://monarchinitiative.org/
BSD 3-Clause "New" or "Revised" License
17 stars 5 forks source link

Add semsimian search call to docker swarm healthcheck #513

Closed kevinschaper closed 10 months ago

kevinschaper commented 10 months ago

I managed to put semsimian-server into some sort of panic state for search (while compare was fine), whatever circumstance causes this is likely to be really tough to find and fix for the semsimian devs, but we should be able to restart the container to bring it back when it happens.

For completeness, here's the error:

monarch-v3_oak.1.p38f6ywphzyq@monarch-v3-dev-api    |    >> Handler search panicked.
monarch-v3_oak.1.p38f6ywphzyq@monarch-v3-dev-api    |    >> A panic is treated as an internal server error.
monarch-v3_oak.1.p38f6ywphzyq@monarch-v3-dev-api    |    >> No 500 catcher registered. Using Rocket default.
monarch-v3_oak.1.p38f6ywphzyq@monarch-v3-dev-api    | thread 'rocket-worker-thread' panicked at src/lib.rs:69:35:
monarch-v3_oak.1.p38f6ywphzyq@monarch-v3-dev-api    | called `Result::unwrap()` on an `Err` value: PoisonError { .. }

I think we're going to want to add two curl commands into the health check in monarch-stack-v3

kevinschaper commented 10 months ago

Completed, and appears to be working well.

The other thing that I added along with this was adding &healthcheck=true to the health check requests, not because the API will do anything with it, but only so that it's easier to filter out healtcheck noise from the log output