search

dandroid126 / lemmy-nhl-gdt-bot

GNU General Public License v3.0
0 stars 2 forks source link

Occasional crashing is occurring when running as a systemd service #19

Closed dandroid126 closed 1 year ago

dandroid126 commented 1 year ago

Every few hours the systemd service restarts. Need to enhance logging to include stdout in the log files to figure out why it is crashing. Need to fail gracefully when an issue occurs.

Also need to tee logs to stdout/stderr, as systemd and podman won't get them otherwise.

dandroid126 commented 1 year ago

Commit 9fbf95101de78bc2ce6b731742cf2dac447264d5 replaces the sleep function with a wait/notify. This will allow us to exit gracefully if a SIGINT or SIGTERM are received. I have a sneaking suspicion that the sleep was related to the crash. Hopefully this fixes it. If not, the same commit adds stdout and stderr to the log files, and vice versa. Hopefully that will give some insight on why it is crashing.

dandroid126 commented 1 year ago

Commit 03bee5e77a923b07368f9c6655e24468ba1643dc fixed a crash coming from requests

dandroid126 commented 1 year ago

Over 24 hours with no crashes. at 48, I will close this issue.

dandroid126 commented 1 year ago

Improved fix by cc27e439eaad26f216ebb1aa74748a449b63d65c and 345d0f061ec16d570daafcb26285e4581821e323. What was happening was sometimes the NHL API wasn't responding for 10+ minutes, causing a timeout. Still don't know why requests throws an exception in that case, but it is being caught now. Also added a timeout of 10 seconds so all the games don't get held up when this happens.

Keep an eye out for game data being empty when this happens. Perhaps there should be a mechanism to signal a failure and skip the post update on this loop instead of returning empty data.