nijel
commented
4 years ago The side effect of this is that threads are crashing when executed concurrently:
dub 28 12:33:10 cachet python[3102]: Exception in thread Thread-8:
dub 28 12:33:10 cachet python[3102]: Traceback (most recent call last):
dub 28 12:33:10 cachet python[3102]: File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
dub 28 12:33:10 cachet python[3102]: self.run()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/cachet_url_monitor/scheduler.py", line 93, in run
dub 28 12:33:10 cachet python[3102]: self.scheduler.start()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/cachet_url_monitor/scheduler.py", line 83, in start
dub 28 12:33:10 cachet python[3102]: schedule.run_pending()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/lib/python3.7/site-packages/schedule/__init__.py", line 563, in run_pending
dub 28 12:33:10 cachet python[3102]: default_scheduler.run_pending()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/lib/python3.7/site-packages/schedule/__init__.py", line 94, in run_pending
dub 28 12:33:10 cachet python[3102]: self._run_job(job)
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/lib/python3.7/site-packages/schedule/__init__.py", line 147, in _run_job
dub 28 12:33:10 cachet python[3102]: ret = job.run()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/lib/python3.7/site-packages/schedule/__init__.py", line 466, in run
dub 28 12:33:10 cachet python[3102]: ret = self.job_func()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/cachet_url_monitor/scheduler.py", line 37, in execute
dub 28 12:33:10 cachet python[3102]: decorator.execute(self.configuration)
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/cachet_url_monitor/scheduler.py", line 62, in execute
dub 28 12:33:10 cachet python[3102]: configuration.push_incident()
dub 28 12:33:10 cachet python[3102]: File "/home/monitor/cachet-url-monitor/cachet_url_monitor/configuration.py", line 256, in push_incident
dub 28 12:33:10 cachet python[3102]: del self.incident_id
dub 28 12:33:10 cachet python[3102]: AttributeError: incident_id
I have configured monitor to monitor 11 services and I've noticed that some of them are triggered several times. Looking at the scheduler module, it doesn't seem to be designed with multiple threads in mind. When triggering it from thread, it can easily happen that more threads get the same pending tasks.
Possible solutions: