search

NHMDenmark / DaSSCo-Integration

This Repo will include integration of dassco storage from northtec
0 stars 0 forks source link

Create health check service and api #89

Open Baeist opened 4 months ago

Baeist commented 4 months ago

We want a service that can keep us updated on, how everything within the integration server is running. This service can be contacted through an api. Each micro service has the responsibility to report errors and warnings to the health service. The health service should not spam us with every instance of errors/warnings that happen in a row. The health service should be responsible for sending out information for issues to both an email and a slack channel to alert us. The health service could run test scripts depending on the errors it receives. The health check service should be listening to the message queue provided by the "health status provider" that is a separate service. The health service would be responsible for slowing down, stopping and starting the other microservices depending on the information it receives.

Baeist commented 3 months ago

Created health api for receiving warnings. This triggers both mails and slack messages indiscriminately.

Baeist commented 3 months ago

Service for mails and slack is set up. Updated to not send mails more often than a configurable time limit. Split warning and error calls. Changes to the run_status of a service should always include a mail - set up run_change_status endpoint. Implemented automatic change to pause status depending on configuration setting. If there are too many errors/time limit it will change to pause status.

Baeist commented 2 weeks ago

HealtApi created a while back. This takes calls from the microservices and determines if a mail/slack message is necessary. It also updates flags of assets with error status if those are included in the call.
Other services would also have the option to pause or change the run status of the micro services. Throttle service fex.