dashboard locks up

[nealw8]

Ben, Back again with another issue. After the first schedule run I found that the dashboard showed that backyard and zone 2 were active in the system status and in the Manual control section. However this was way after the end of the scheduled watering. Upon checking outside all zones were inactive. Further if I clicked on the stop schedule tab nothing happen for about 30 secs then I got a 502 Bad Gateway and the dashboard disappeared. The pi was still running because I could SSH in. I tried rebooting and a total shutdown for over night and the dashboard would come back but it did not rectify the problem of the active schedules or zones.

Neal

[nealw8]

It's hanging up in the manualcontrol section because I can get back into the dashboard by re-entering the 192.168.0.103 without the \manualcontrol following.

[nebhead]

Can you press F12 in your browser when this happens and look at the console tab. Let me know if you see any errors there.

[nealw8]

On my computer the F12 button is the airplane mode activation and nothing happens

[nealw8]

here is the events log: nealw@irrigator:/usr/local/bin/irrigator $ cat events.log 2024-06-01 17:55:46 Control Script Starting 2024-06-01 17:55:46 Initializing Relays on Raspberry Pi. 2024-06-01 17:55:46 Current Temperature: 81F Precipitation History: 0.0 inches Precipitation Forecast: 0.0 inches 2024-06-01 17:55:46 Force run selected. Ignoring weather. 2024-06-01 17:55:46 Manual Run Selected. 2024-06-01 17:55:46 Turning on Zone: zone_02 for 10 minutes. 2024-06-01 17:56:28 Turning off Zone: zone_02. 2024-06-01 17:56:28 Exiting with errorcode = 42 2024-06-01 17:56:28 Control Script Ended 2024-06-02 00:30:03 Control Script Starting 2024-06-02 00:30:03 Initializing Relays on Raspberry Pi. 2024-06-02 00:30:03 Current Temperature: 64F Precipitation History: 0.0 inches Precipitation Forecast: 0.0 inches 2024-06-02 00:30:03 Schedule Run Selected with Schedule: BackYard 2024-06-02 00:30:03 BackYard found in JSON file. Running Now. 2024-06-02 00:30:03 Turning on Zone: zone_01 for 60 minutes. 2024-06-02 01:30:04 Turning off Zone: zone_01. 2024-06-02 01:30:04 Turning on Zone: zone_02 for 60 minutes. 2024-06-02 02:30:05 Turning off Zone: zone_02. 2024-06-02 02:30:05 Turning on Zone: zone_03 for 60 minutes. 2024-06-02 03:30:05 Turning off Zone: zone_03. 2024-06-02 03:30:05 Exiting with errorcode = 0 2024-06-02 03:30:05 Control Script Ended 2024-06-02 04:30:03 Control Script Starting 2024-06-02 04:30:03 Initializing Relays on Raspberry Pi. 2024-06-02 04:30:03 Current Temperature: 60F Precipitation History: 0.0 inches Precipitation Forecast: 0.0 inches 2024-06-02 04:30:03 Schedule Run Selected with Schedule: FrontYard 2024-06-02 04:30:03 FrontYard found in JSON file. Running Now. 2024-06-02 04:30:03 Turning on Zone: zone_01 for 30 minutes. 2024-06-02 05:00:04 Turning off Zone: zone_01. 2024-06-02 05:00:04 Turning on Zone: zone_02 for 30 minutes. 2024-06-02 05:30:05 Turning off Zone: zone_02. 2024-06-02 05:30:05 Turning on Zone: zone_03 for 30 minutes. 2024-06-02 06:00:06 Turning off Zone: zone_03. 2024-06-02 06:00:06 Exiting with errorcode = 0 2024-06-02 06:00:06 Control Script Ended 2024-06-02 14:59:39 Admin: Reboot 2024-06-02 15:00:35 Control Script Starting 2024-06-02 15:00:35 Initializing Relays on Raspberry Pi. 2024-06-02 15:00:36 Current Temperature: 81F Precipitation History: 0.0 inches Precipitation Forecast: 0.04 inches 2024-06-02 15:00:36 Initialize Relays Selected. 2024-06-02 15:00:36 Exiting with errorcode = 0 2024-06-02 15:00:36 Control Script Ended 2024-06-02 15:01:26 Admin: Reboot 2024-06-02 15:02:23 Control Script Starting 2024-06-02 15:02:25 Initializing Relays on Raspberry Pi. 2024-06-02 15:02:25 Current Temperature: 81F Precipitation History: 0.0 inches Precipitation Forecast: 0.04 inches 2024-06-02 15:02:25 Initialize Relays Selected. 2024-06-02 15:02:25 Exiting with errorcode = 0 2024-06-02 15:02:25 Control Script Ended 2024-06-02 15:05:26 Admin: Reboot 2024-06-02 15:05:30 Admin: Reboot 2024-06-02 15:06:19 Control Script Starting 2024-06-02 15:06:19 Initializing Relays on Raspberry Pi. 2024-06-02 15:06:20 Current Temperature: 81F Precipitation History: 0.0 inches Precipitation Forecast: 0.04 inches 2024-06-02 15:06:20 Initialize Relays Selected. 2024-06-02 15:06:20 Exiting with errorcode = 0 2024-06-02 15:06:20 Control Script Ended 2024-06-02 15:11:32 Admin: Shutdown 2024-06-02 15:12:25 Control Script Starting 2024-06-02 15:12:26 Initializing Relays on Raspberry Pi. 2024-06-02 15:12:26 Current Temperature: 81F Precipitation History: 0.0 inches Precipitation Forecast: 0.04 inches 2024-06-02 15:12:26 Initialize Relays Selected. 2024-06-02 15:12:26 Exiting with errorcode = 0 2024-06-02 15:12:26 Control Script Ended 2024-06-02 15:22:04 Admin: Shutdown 2024-06-02 15:22:05 Admin: Shutdown 2024-06-02 15:22:58 Control Script Starting 2024-06-02 15:22:59 Initializing Relays on Raspberry Pi. 2024-06-02 15:22:59 Current Temperature: 81F Precipitation History: 0.0 inches Precipitation Forecast: 0.04 inches 2024-06-02 15:22:59 Initialize Relays Selected. 2024-06-02 15:22:59 Exiting with errorcode = 0 2024-06-02 15:22:59 Control Script Ended 2024-06-03 11:47:32 Control Script Starting 2024-06-03 11:47:33 Initializing Relays on Raspberry Pi. 2024-06-03 11:47:33 Current Temperature: 78F Precipitation History: 0.0 inches Precipitation Forecast: 0.13 inches 2024-06-03 11:47:33 Initialize Relays Selected. 2024-06-03 11:47:33 Exiting with errorcode = 0 2024-06-03 11:47:33 Control Script Ended 2024-06-03 12:24:09 Control Script Starting 2024-06-03 12:24:10 Initializing Relays on Raspberry Pi. 2024-06-03 12:24:10 Current Temperature: 80F Precipitation History: 0.0 inches Precipitation Forecast: 0.13 inches 2024-06-03 12:24:10 Force run selected. Ignoring weather. 2024-06-03 12:24:10 Schedule Run Selected with Schedule: FrontYard 2024-06-03 12:24:10 FrontYard found in JSON file. Running Now. 2024-06-03 12:24:10 Turning on Zone: zone_01 for 30 minutes. 2024-06-03 12:24:10 Turning off Zone: zone_01. 2024-06-03 12:24:10 Turning on Zone: zone_02 for 30 minutes. 2024-06-03 12:24:10 Turning off Zone: zone_02. 2024-06-03 12:24:10 Turning on Zone: zone_03 for 30 minutes. 2024-06-03 12:24:10 Turning off Zone: zone_03. 2024-06-03 12:24:10 Exiting with errorcode = 42 2024-06-03 12:24:10 Control Script Ended

[nebhead]

On my computer the F12 button is the airplane mode activation and nothing happens

You might have to press 'function' + 'F12' in your browser. Alternatively, you can go into your browser settings and look for developer tools or browser console. The log that you have above looks all good, so no issues with that. It's just the web interface that is not really behaving.

[nealw8]

console tab empty

[nealw8]

It is browser independent. Happens with chrome. Brave firefox.

Found this in firefox This page is in Quirks Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. manualcontrol

​

[nealw8]

and this in brave by going in with developer tools: One or more documents in this page is in Quirks Mode, which will render the affected document(s) with quirks incompatible with the current HTML and CSS specifications. Quirks Mode exists mostly due to historical reasons. If this is not intentional, you can add or modify the DOCTYPE to be <!DOCTYPE html> to render the page in No Quirks Mode. ​

[nebhead]

On the admin page, try clearing the controls:

This won't affect other settings. But it should clear out and errant status data that might be there.

[nealw8]

Thanks, that did it. What do you suppose instigated that problem?

[nebhead]

Thanks, that did it. What do you suppose instigated that problem?

Honestly, I'm not really sure what caused that to happen. It could be that there was a read/write collision where the control script was trying to write the status to the file and the webapp script was reading at the same time. Hopefully this doesn't happen again, but in any case you know what to do if it does.

[nealw8]

Thanks again

[nealw8]

Bug still there. After today's schedule run the same result as above except a different button locked. Cleared it as you suggested. Unsure if I leave it whether the next cycle will run.

[nebhead]

Maybe we can check the log files here:

cd /usr/local/bin/irrigator/logs/
cat webapp.err.log
cat webapp.out.log 

That might tell us if there was read/write collision showing up.

[nealw8]

nealw@irrigator:/usr/local/bin/irrigator/logs $ cat webapp.err.log [2024-05-18 13:28:42 -0400] [776] [INFO] Starting gunicorn 20.1.0 [2024-05-18 13:28:42 -0400] [776] [INFO] Listening at: http://127.0.0.1:8000 (776) [2024-05-18 13:28:42 -0400] [776] [INFO] Using worker: sync [2024-05-18 13:28:42 -0400] [786] [INFO] Booting worker with pid: 786 [2024-05-18 14:29:45 -0400] [776] [INFO] Handling signal: term [2024-05-18 14:29:45 -0400] [786] [INFO] Worker exiting (pid: 786) [2024-05-18 14:29:45 -0400] [776] [INFO] Shutting down: Master [2024-05-18 14:29:54 -0400] [1185] [INFO] Starting gunicorn 20.1.0 [2024-05-18 14:29:54 -0400] [1185] [INFO] Listening at: http://127.0.0.1:8000 (1185) [2024-05-18 14:29:54 -0400] [1185] [INFO] Using worker: sync [2024-05-18 14:29:54 -0400] [1186] [INFO] Booting worker with pid: 1186 [2024-05-18 16:44:32 -0400] [1185] [INFO] Handling signal: term [2024-05-18 16:44:32 -0400] [1186] [INFO] Worker exiting (pid: 1186) [2024-05-18 16:44:33 -0400] [1185] [INFO] Shutting down: Master [2024-05-18 16:45:54 -0400] [774] [INFO] Starting gunicorn 20.1.0 [2024-05-18 16:45:54 -0400] [774] [INFO] Listening at: http://127.0.0.1:8000 (774) [2024-05-18 16:45:54 -0400] [774] [INFO] Using worker: sync [2024-05-18 16:45:54 -0400] [794] [INFO] Booting worker with pid: 794 [2024-05-18 18:21:16 -0400] [774] [CRITICAL] WORKER TIMEOUT (pid:794) [2024-05-18 18:21:16 -0400] [794] [INFO] Worker exiting (pid: 794) [2024-05-18 18:21:17 -0400] [1080] [INFO] Booting worker with pid: 1080 [2024-05-18 18:44:00 -0400] [774] [INFO] Handling signal: term [2024-05-18 18:44:00 -0400] [1080] [INFO] Worker exiting (pid: 1080) [2024-05-18 18:44:01 -0400] [774] [INFO] Shutting down: Master [2024-05-18 18:45:21 -0400] [758] [INFO] Starting gunicorn 20.1.0 [2024-05-18 18:45:21 -0400] [758] [INFO] Listening at: http://127.0.0.1:8000 (758) [2024-05-18 18:45:21 -0400] [758] [INFO] Using worker: sync [2024-05-18 18:45:21 -0400] [792] [INFO] Booting worker with pid: 792 [2024-05-19 09:37:47 -0400] [758] [CRITICAL] WORKER TIMEOUT (pid:792) [2024-05-19 09:37:48 -0400] [792] [INFO] Worker exiting (pid: 792) [2024-05-19 09:37:49 -0400] [758] [WARNING] Worker with pid 792 was terminated due to signal 9 [2024-05-19 09:37:49 -0400] [800] [INFO] Booting worker with pid: 800 [2024-05-19 09:38:13 -0400] [758] [INFO] Handling signal: term [2024-05-19 09:38:13 -0400] [800] [INFO] Worker exiting (pid: 800) [2024-05-19 09:38:13 -0400] [758] [INFO] Shutting down: Master [2024-05-19 09:39:30 -0400] [769] [INFO] Starting gunicorn 20.1.0 [2024-05-19 09:39:30 -0400] [769] [INFO] Listening at: http://127.0.0.1:8000 (769) [2024-05-19 09:39:30 -0400] [769] [INFO] Using worker: sync [2024-05-19 09:39:30 -0400] [791] [INFO] Booting worker with pid: 791 [2024-05-19 09:40:56 -0400] [769] [CRITICAL] WORKER TIMEOUT (pid:791) [2024-05-19 09:40:56 -0400] [791] [INFO] Worker exiting (pid: 791) [2024-05-19 09:40:57 -0400] [792] [INFO] Booting worker with pid: 792 [2024-05-19 10:47:33 -0400] [769] [INFO] Handling signal: term [2024-05-19 10:47:33 -0400] [792] [INFO] Worker exiting (pid: 792) [2024-05-19 10:47:34 -0400] [769] [INFO] Shutting down: Master [2024-05-19 10:48:50 -0400] [770] [INFO] Starting gunicorn 20.1.0 [2024-05-19 10:48:50 -0400] [770] [INFO] Listening at: http://127.0.0.1:8000 (770) [2024-05-19 10:48:50 -0400] [770] [INFO] Using worker: sync [2024-05-19 10:48:50 -0400] [788] [INFO] Booting worker with pid: 788 [2024-05-19 11:56:25 -0400] [776] [INFO] Starting gunicorn 20.1.0 [2024-05-19 11:56:25 -0400] [776] [INFO] Listening at: http://127.0.0.1:8000 (776) [2024-05-19 11:56:25 -0400] [776] [INFO] Using worker: sync [2024-05-19 11:56:25 -0400] [789] [INFO] Booting worker with pid: 789 [2024-05-19 12:17:07 -0400] [776] [CRITICAL] WORKER TIMEOUT (pid:789) [2024-05-19 12:17:07 -0400] [789] [INFO] Worker exiting (pid: 789) [2024-05-19 12:17:08 -0400] [776] [WARNING] Worker with pid 789 was terminated due to signal 9 [2024-05-19 12:17:08 -0400] [799] [INFO] Booting worker with pid: 799 [2024-05-19 12:18:50 -0400] [773] [INFO] Starting gunicorn 20.1.0 [2024-05-19 12:18:50 -0400] [773] [INFO] Listening at: http://127.0.0.1:8000 (773) [2024-05-19 12:18:50 -0400] [773] [INFO] Using worker: sync [2024-05-19 12:18:50 -0400] [791] [INFO] Booting worker with pid: 791 [2024-05-19 13:36:15 -0400] [773] [INFO] Handling signal: term [2024-05-19 13:36:15 -0400] [791] [INFO] Worker exiting (pid: 791) [2024-05-19 13:36:16 -0400] [773] [INFO] Shutting down: Master [2024-05-19 13:37:32 -0400] [775] [INFO] Starting gunicorn 20.1.0 [2024-05-19 13:37:32 -0400] [775] [INFO] Listening at: http://127.0.0.1:8000 (775) [2024-05-19 13:37:32 -0400] [775] [INFO] Using worker: sync [2024-05-19 13:37:32 -0400] [790] [INFO] Booting worker with pid: 790 [2024-05-19 13:47:47 -0400] [775] [CRITICAL] WORKER TIMEOUT (pid:790) [2024-05-19 13:47:47 -0400] [790] [INFO] Worker exiting (pid: 790) [2024-05-19 13:47:48 -0400] [791] [INFO] Booting worker with pid: 791 [2024-05-19 14:13:22 -0400] [775] [INFO] Handling signal: term [2024-05-19 14:13:22 -0400] [791] [INFO] Worker exiting (pid: 791) [2024-05-19 14:13:22 -0400] [775] [INFO] Shutting down: Master [2024-05-19 14:14:38 -0400] [769] [INFO] Starting gunicorn 20.1.0 [2024-05-19 14:14:38 -0400] [769] [INFO] Listening at: http://127.0.0.1:8000 (769) [2024-05-19 14:14:38 -0400] [769] [INFO] Using worker: sync [2024-05-19 14:14:38 -0400] [788] [INFO] Booting worker with pid: 788 [2024-05-20 09:32:52 -0400] [769] [CRITICAL] WORKER TIMEOUT (pid:788) [2024-05-20 09:32:52 -0400] [788] [INFO] Worker exiting (pid: 788) [2024-05-20 09:32:53 -0400] [769] [WARNING] Worker with pid 788 was terminated due to signal 9 [2024-05-20 09:32:53 -0400] [795] [INFO] Booting worker with pid: 795 [2024-05-20 10:31:53 -0400] [769] [INFO] Handling signal: term [2024-05-20 10:31:53 -0400] [795] [INFO] Worker exiting (pid: 795) [2024-05-20 10:31:54 -0400] [769] [INFO] Shutting down: Master [2024-05-20 10:33:09 -0400] [762] [INFO] Starting gunicorn 20.1.0 [2024-05-20 10:33:09 -0400] [762] [INFO] Listening at: http://127.0.0.1:8000 (762) [2024-05-20 10:33:09 -0400] [762] [INFO] Using worker: sync [2024-05-20 10:33:09 -0400] [787] [INFO] Booting worker with pid: 787 [2024-05-20 11:13:32 -0400] [762] [CRITICAL] WORKER TIMEOUT (pid:787) [2024-05-20 11:13:32 -0400] [787] [INFO] Worker exiting (pid: 787) [2024-05-20 11:13:33 -0400] [788] [INFO] Booting worker with pid: 788 [2024-06-02 08:49:37 -0400] [762] [CRITICAL] WORKER TIMEOUT (pid:788) [2024-06-02 08:49:37 -0400] [788] [INFO] Worker exiting (pid: 788) [2024-06-02 08:49:37 -0400] [24954] [INFO] Booting worker with pid: 24954 [2024-06-02 08:50:11 -0400] [762] [CRITICAL] WORKER TIMEOUT (pid:24954) [2024-06-02 08:50:11 -0400] [24954] [INFO] Worker exiting (pid: 24954) [2024-06-02 08:50:12 -0400] [24957] [INFO] Booting worker with pid: 24957 [2024-06-02 09:06:20 -0400] [762] [CRITICAL] WORKER TIMEOUT (pid:24957) [2024-06-02 09:06:20 -0400] [24957] [INFO] Worker exiting (pid: 24957) [2024-06-02 09:06:20 -0400] [24969] [INFO] Booting worker with pid: 24969 [2024-06-02 09:06:53 -0400] [762] [CRITICAL] WORKER TIMEOUT (pid:24969) [2024-06-02 09:06:53 -0400] [24969] [INFO] Worker exiting (pid: 24969) [2024-06-02 09:06:54 -0400] [24970] [INFO] Booting worker with pid: 24970 [2024-06-02 09:07:27 -0400] [762] [CRITICAL] WORKER TIMEOUT (pid:24970) [2024-06-02 09:07:27 -0400] [24970] [INFO] Worker exiting (pid: 24970) [2024-06-02 09:07:27 -0400] [24971] [INFO] Booting worker with pid: 24971 [2024-06-02 14:59:58 -0400] [762] [INFO] Handling signal: term [2024-06-02 14:59:58 -0400] [24971] [INFO] Worker exiting (pid: 24971) [2024-06-02 14:59:59 -0400] [762] [INFO] Shutting down: Master [2024-06-02 15:01:15 -0400] [774] [INFO] Starting gunicorn 20.1.0 [2024-06-02 15:01:15 -0400] [774] [INFO] Listening at: http://127.0.0.1:8000 (774) [2024-06-02 15:01:15 -0400] [774] [INFO] Using worker: sync [2024-06-02 15:01:15 -0400] [788] [INFO] Booting worker with pid: 788 [2024-06-02 15:01:46 -0400] [774] [INFO] Handling signal: term [2024-06-02 15:01:46 -0400] [788] [INFO] Worker exiting (pid: 788) [2024-06-02 15:01:47 -0400] [774] [INFO] Shutting down: Master [2024-06-02 15:03:03 -0400] [775] [INFO] Starting gunicorn 20.1.0 [2024-06-02 15:03:03 -0400] [775] [INFO] Listening at: http://127.0.0.1:8000 (775) [2024-06-02 15:03:03 -0400] [775] [INFO] Using worker: sync [2024-06-02 15:03:03 -0400] [785] [INFO] Booting worker with pid: 785 [2024-06-02 15:04:01 -0400] [775] [CRITICAL] WORKER TIMEOUT (pid:785) [2024-06-02 15:04:01 -0400] [785] [INFO] Worker exiting (pid: 785) [2024-06-02 15:04:02 -0400] [953] [INFO] Booting worker with pid: 953 [2024-06-02 15:05:43 -0400] [775] [INFO] Handling signal: term [2024-06-02 15:05:43 -0400] [953] [INFO] Worker exiting (pid: 953) [2024-06-02 15:05:44 -0400] [775] [INFO] Shutting down: Master [2024-06-02 15:07:00 -0400] [756] [INFO] Starting gunicorn 20.1.0 [2024-06-02 15:07:00 -0400] [756] [INFO] Listening at: http://127.0.0.1:8000 (756) [2024-06-02 15:07:00 -0400] [756] [INFO] Using worker: sync [2024-06-02 15:07:00 -0400] [786] [INFO] Booting worker with pid: 786 [2024-06-02 15:08:30 -0400] [756] [CRITICAL] WORKER TIMEOUT (pid:786) [2024-06-02 15:08:30 -0400] [786] [INFO] Worker exiting (pid: 786) [2024-06-02 15:08:31 -0400] [1082] [INFO] Booting worker with pid: 1082 [2024-06-02 15:09:05 -0400] [756] [CRITICAL] WORKER TIMEOUT (pid:1082) [2024-06-02 15:09:05 -0400] [1082] [INFO] Worker exiting (pid: 1082) [2024-06-02 15:09:05 -0400] [1083] [INFO] Booting worker with pid: 1083 [2024-06-02 15:09:38 -0400] [756] [CRITICAL] WORKER TIMEOUT (pid:1083) [2024-06-02 15:09:38 -0400] [1083] [INFO] Worker exiting (pid: 1083) [2024-06-02 15:09:39 -0400] [1084] [INFO] Booting worker with pid: 1084 [2024-06-02 15:10:12 -0400] [756] [CRITICAL] WORKER TIMEOUT (pid:1084) [2024-06-02 15:10:12 -0400] [1084] [INFO] Worker exiting (pid: 1084) [2024-06-02 15:10:12 -0400] [1085] [INFO] Booting worker with pid: 1085 [2024-06-02 15:10:46 -0400] [756] [CRITICAL] WORKER TIMEOUT (pid:1085) [2024-06-02 15:10:46 -0400] [1085] [INFO] Worker exiting (pid: 1085) [2024-06-02 15:10:46 -0400] [1086] [INFO] Booting worker with pid: 1086 [2024-06-02 15:11:19 -0400] [756] [CRITICAL] WORKER TIMEOUT (pid:1086) [2024-06-02 15:11:19 -0400] [1086] [INFO] Worker exiting (pid: 1086) [2024-06-02 15:11:20 -0400] [1090] [INFO] Booting worker with pid: 1090 [2024-06-02 15:11:50 -0400] [756] [INFO] Handling signal: term [2024-06-02 15:11:50 -0400] [1090] [INFO] Worker exiting (pid: 1090) [2024-06-02 15:11:50 -0400] [756] [INFO] Shutting down: Master [2024-06-02 15:13:05 -0400] [766] [INFO] Starting gunicorn 20.1.0 [2024-06-02 15:13:05 -0400] [766] [INFO] Listening at: http://127.0.0.1:8000 (766) [2024-06-02 15:13:05 -0400] [766] [INFO] Using worker: sync [2024-06-02 15:13:05 -0400] [786] [INFO] Booting worker with pid: 786 [2024-06-02 15:19:09 -0400] [766] [CRITICAL] WORKER TIMEOUT (pid:786) [2024-06-02 15:19:09 -0400] [786] [INFO] Worker exiting (pid: 786) [2024-06-02 15:19:10 -0400] [788] [INFO] Booting worker with pid: 788 [2024-06-02 15:20:44 -0400] [766] [CRITICAL] WORKER TIMEOUT (pid:788) [2024-06-02 15:20:44 -0400] [788] [INFO] Worker exiting (pid: 788) [2024-06-02 15:20:45 -0400] [1083] [INFO] Booting worker with pid: 1083 [2024-06-02 15:21:20 -0400] [766] [CRITICAL] WORKER TIMEOUT (pid:1083) [2024-06-02 15:21:20 -0400] [1083] [INFO] Worker exiting (pid: 1083) [2024-06-02 15:21:20 -0400] [1084] [INFO] Booting worker with pid: 1084 [2024-06-02 15:21:53 -0400] [766] [CRITICAL] WORKER TIMEOUT (pid:1084) [2024-06-02 15:21:53 -0400] [1084] [INFO] Worker exiting (pid: 1084) [2024-06-02 15:21:54 -0400] [1085] [INFO] Booting worker with pid: 1085 [2024-06-02 15:22:22 -0400] [766] [INFO] Handling signal: term [2024-06-02 15:22:22 -0400] [1085] [INFO] Worker exiting (pid: 1085) [2024-06-02 15:22:23 -0400] [766] [INFO] Shutting down: Master [2024-06-02 15:23:40 -0400] [771] [INFO] Starting gunicorn 20.1.0 [2024-06-02 15:23:40 -0400] [771] [INFO] Listening at: http://127.0.0.1:8000 (771) [2024-06-02 15:23:40 -0400] [771] [INFO] Using worker: sync [2024-06-02 15:23:40 -0400] [787] [INFO] Booting worker with pid: 787 [2024-06-02 18:34:24 -0400] [771] [CRITICAL] WORKER TIMEOUT (pid:787) [2024-06-02 18:34:24 -0400] [787] [INFO] Worker exiting (pid: 787) [2024-06-02 18:34:25 -0400] [771] [WARNING] Worker with pid 787 was terminated due to signal 9 [2024-06-02 18:34:26 -0400] [790] [INFO] Booting worker with pid: 790 [2024-06-02 18:39:31 -0400] [771] [CRITICAL] WORKER TIMEOUT (pid:790) [2024-06-02 18:39:31 -0400] [790] [INFO] Worker exiting (pid: 790) [2024-06-02 18:39:32 -0400] [1088] [INFO] Booting worker with pid: 1088 [2024-06-02 18:51:11 -0400] [771] [CRITICAL] WORKER TIMEOUT (pid:1088) [2024-06-02 18:51:11 -0400] [1088] [INFO] Worker exiting (pid: 1088) [2024-06-02 18:51:11 -0400] [1101] [INFO] Booting worker with pid: 1101 [2024-06-02 18:51:44 -0400] [771] [CRITICAL] WORKER TIMEOUT (pid:1101) [2024-06-02 18:51:44 -0400] [1101] [INFO] Worker exiting (pid: 1101) [2024-06-02 18:51:45 -0400] [1102] [INFO] Booting worker with pid: 1102 [2024-06-03 07:54:10 -0400] [771] [CRITICAL] WORKER TIMEOUT (pid:1102) [2024-06-03 07:54:10 -0400] [1102] [INFO] Worker exiting (pid: 1102) [2024-06-03 07:54:10 -0400] [2081] [INFO] Booting worker with pid: 2081 [2024-06-03 11:29:39 -0400] [771] [CRITICAL] WORKER TIMEOUT (pid:2081) [2024-06-03 11:29:39 -0400] [2081] [INFO] Worker exiting (pid: 2081) [2024-06-03 11:29:39 -0400] [2341] [INFO] Booting worker with pid: 2341 [2024-06-03 11:46:55 -0400] [771] [INFO] Handling signal: term [2024-06-03 11:46:55 -0400] [2341] [INFO] Worker exiting (pid: 2341) [2024-06-03 11:46:56 -0400] [771] [INFO] Shutting down: Master [2024-06-03 11:48:13 -0400] [767] [INFO] Starting gunicorn 20.1.0 [2024-06-03 11:48:13 -0400] [767] [INFO] Listening at: http://127.0.0.1:8000 (767) [2024-06-03 11:48:13 -0400] [767] [INFO] Using worker: sync [2024-06-03 11:48:13 -0400] [789] [INFO] Booting worker with pid: 789 [2024-06-03 12:01:41 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:789) [2024-06-03 12:01:41 -0400] [789] [INFO] Worker exiting (pid: 789) [2024-06-03 12:01:41 -0400] [1116] [INFO] Booting worker with pid: 1116 [2024-06-03 12:24:39 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1116) [2024-06-03 12:24:39 -0400] [1116] [INFO] Worker exiting (pid: 1116) [2024-06-03 12:24:39 -0400] [1145] [INFO] Booting worker with pid: 1145 [2024-06-03 12:30:04 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1145) [2024-06-03 12:30:04 -0400] [1145] [INFO] Worker exiting (pid: 1145) [2024-06-03 12:30:05 -0400] [1156] [INFO] Booting worker with pid: 1156 [2024-06-03 12:32:36 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1156) [2024-06-03 12:32:36 -0400] [1156] [INFO] Worker exiting (pid: 1156) [2024-06-03 12:32:36 -0400] [1157] [INFO] Booting worker with pid: 1157 [2024-06-03 12:33:10 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1157) [2024-06-03 12:33:10 -0400] [1157] [INFO] Worker exiting (pid: 1157) [2024-06-03 12:33:10 -0400] [1158] [INFO] Booting worker with pid: 1158 [2024-06-03 12:54:33 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1158) [2024-06-03 12:54:33 -0400] [1158] [INFO] Worker exiting (pid: 1158) [2024-06-03 12:54:34 -0400] [1186] [INFO] Booting worker with pid: 1186 [2024-06-03 13:20:37 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1186) [2024-06-03 13:20:37 -0400] [1186] [INFO] Worker exiting (pid: 1186) [2024-06-03 13:20:38 -0400] [1216] [INFO] Booting worker with pid: 1216 [2024-06-03 13:21:35 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1216) [2024-06-03 13:21:35 -0400] [1216] [INFO] Worker exiting (pid: 1216) [2024-06-03 13:21:35 -0400] [1217] [INFO] Booting worker with pid: 1217 [2024-06-04 13:56:21 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:1217) [2024-06-04 13:56:21 -0400] [1217] [INFO] Worker exiting (pid: 1217) [2024-06-04 13:56:22 -0400] [3100] [INFO] Booting worker with pid: 3100 [2024-06-04 16:08:32 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:3100) [2024-06-04 16:08:32 -0400] [3100] [INFO] Worker exiting (pid: 3100) [2024-06-04 16:08:32 -0400] [3251] [INFO] Booting worker with pid: 3251 [2024-06-04 16:09:29 -0400] [767] [CRITICAL] WORKER TIMEOUT (pid:3251) [2024-06-04 16:09:29 -0400] [3251] [INFO] Worker exiting (pid: 3251) [2024-06-04 16:09:29 -0400] [3255] [INFO] Booting worker with pid: 3255

[nebhead]

Thanks for posting your logs. Unfortunately, I'm not getting any clues on why the data is getting corrupted. Maybe we can instrument the code to see if the writes are getting blocked somehow. Let me get back to you on this. In the meantime, I may delete your above posts to protect your personal information.