search

jhuckaby / Cronicle

A simple, distributed task scheduler and runner with a web based UI.
http://cronicle.net
Other
3.93k stars 391 forks source link

Cronicle start problem - PID incorrectly attributed - EASY FIX #789

Closed cliffshaw closed 3 months ago

cliffshaw commented 3 months ago

Summary

TLDR: Cronicle wrongly thinks it's running when it's not.

This screenshot says it all. Cronicle thinks it's running because another process is running that has the stale PID.

image

This machine has only been live for 3 hours due to a reboot. Yet the PID file it wrote was from yesterday.

Steps to reproduce the problem

Steal a PID from any running job. Cronicle will not check whether this PID is stale and actually taken up by another process.

Operating system and version?

Ubuntu 22.04

Node.js version?

20 something.

Cronicle software version?

0.9.53

Are you using a multi-server setup, or just a single server?

Multi

jhuckaby commented 3 months ago

Thank you for reporting this. I'll fix it ASAP.

cliffshaw commented 3 months ago

With pleasure... brilliant creation and thank you for your efforts on this.

jhuckaby commented 3 months ago

Fixed in v0.9.57.

I just want to point out that if you encounter a situation where the Cronicle PID file is hanging around on your filesystem, and it exists when it tries to start back up, this is VERY BAD. It's a situation which should "never" occur during normal operations, because Cronicle ALWAYS deletes its PID file on normal shutdown. So a stale PID file can only occur is if Cronicle crashed, or if the server / VM / container died while it was running. This is a terrible situation, because Cronicle's internal database can become corrupted. (

(Note that it ATTEMPTS to rollback the database on the next startup after an unclean shutdown, but it's generally a situation you don't want to encounter.)

So while the "bug" is now fixed in v0.9.57 and Cronicle will now correctly sniff the PID process and detect if it is really Cronicle or some other process, the fact that you have a stale PID file that now points to your [nfsd] daemon means that your Cronicle crashed horribly, or your server powered off unexpectedly, or you had a kernel panic. Either way, this makes Cronicle very sad, similar to powering off a database while it's writing records.

Just wanted to point that out. Thanks again for the issue report!

cliffshaw commented 3 months ago

Thanks for the speedy fix on this.

Just to make you aware, this was a Worker machine and not the Cronicle server itself. Unexpected reboots are part of life so I'm glad to hear that you've got some protections in place if the server were to unexpectedly reboot.

jhuckaby commented 3 months ago

Ah good, I'm glad it was just a worker. Those don't write any data to the internal database, so you're good.

Yup, unexpected reboots do happen 😊