Jenkins check shows passing even though the status is failed

arachnys / cabot

Self-hosted, easily-deployable monitoring and alerts service - like a lightweight PagerDuty

MIT License

5.59k stars 593 forks source link

Jenkins check shows passing even though the status is failed #695

Open jmbeach opened 4 years ago

jmbeach commented 4 years ago

This isn't a duplicate of #505

Somehow the check fails, but it doesn't change the status. It's communicating with jenkins successfully.

dbuxton commented 4 years ago

My guess is that this is a consistency issue caused by the async task that updates the service status either not running, or being executed with a significant delay. If you click the refresh button on the service page (the one in your first screenshot), does the status update correctly?

You might want to look at task logs to see if there are things stuck in the queue

jmbeach commented 4 years ago

It doesn't update. I'm just running everything from docker. Is there a troubleshooting guide? I'm not sure if by task log you just mean the logs of the containers.

jmbeach commented 4 years ago

HTTP checks work / send e-mail when failing but not jenkins

dbuxton commented 4 years ago

Can you allocate more resources? This sounds like a docker config error to me.

Yes, by task logs I mean logs of the worker processes. You might want to try playing with machine resources - it sounds like you might be overloading your containers?

jmbeach commented 4 years ago

Cool. I'll look into that. Something to note is I've been trying to run this on Windows, but I'm switching to trying to run it on Ubuntu next just because docker is way easier on linux. If I have the same issue, I'm going to try to run all of the containers except for the worker and then debug the worker in pycharm and see if I find any errors.