search

jhuckaby / Cronicle

A simple, distributed task scheduler and runner with a web based UI.
http://cronicle.net
Other
3.74k stars 385 forks source link

Log watcher Error #610

Open sudhakovid opened 1 year ago

sudhakovid commented 1 year ago

Summary

Receiving log watcher errors while running a job.

Steps to reproduce the problem

Your Setup

Operating system and version?

Centos 7.4

Node.js version?

16.20.0

Cronicle software version?

0.9.21

Are you using a multi-server setup, or just a single server?

Standalone

Are you using the filesystem as back-end storage, or S3/Couchbase?

s3

Can you reproduce the crash consistently?

yes

Log Excerpts

Log Watcher: Connecting to server: https://cronicle.my-website.com/...

Log Watcher: Server Connect Error: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client, but they are not compatible (more information here: https://socket.io/docs/v3/migrating-from-2-x-to-3-0/) (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client, but they are not compatible (more information here: https://socket.io/docs/v3/migrating-from-2-x-to-3-0/) (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client, but they are not compatible (more information here: https://socket.io/docs/v3/migrating-from-2-x-to-3-0/) (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client, but they are not compatible (more information here: https://socket.io/docs/v3/migrating-from-2-x-to-3-0/) (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client, but they are not compatible (more information here: https://socket.io/docs/v3/migrating-from-2-x-to-3-0/) (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client, but they are not compatible (more information here: https://socket.io/docs/v3/migrating-from-2-x-to-3-0/) (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: websocket error (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: websocket error (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: websocket error (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: websocket error (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: websocket error (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: timeout (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: websocket error (https://cronicle.my-website.com/)

Log Watcher: Server Connect Error: Error: timeout (https://cronicle.my-website.com/)

jhuckaby commented 1 year ago

My guess is that your worker servers are running an older version of Cronicle. Make sure they are all running the same version, and this error should go away.

However, in your issue report you said you are running a single standalone server. Is that true? If so, then I really cannot explain this error at all. The Socket.IO libraries are shared between the client and server, so they should never become out of date like this. Extremely odd.

Delete and reinstall Cronicle from scratch I guess?

sudhakovid commented 1 year ago

Thanks @jhuckaby. Yes, we are actually running a standalone master node for cronicle. We just upgrade from 0.8.x to 0.9.21 and it's entirely different server/installation/configuration. All the jobs are running fine but the only thing is we are getting to see these log watcher errors but the job succeeds which is weird.

jhuckaby commented 1 year ago

Yup, that's very bizarre indeed. I cannot fathom what would cause that, but I will say, the two versions you mentioned (upgrading from 0.8.x to 0.9.21) will cause Socket.IO to perform a major non-compatible upgrade. It is possible that the upgrade didn't finish cleanly, and part of the distribution is still left with the old version. You might want to consider wiping and reinstalling clean.

You could also try clearing your browser's cache. Perhaps the old 0.8.x Socket.IO client libraries are stuck in cache somewhere.

Hahlh commented 1 year ago

Experiencing the same with version 0.9.25 via docker compose (https://github.com/bluet/docker-cronicle-docker).

Also, it seems that node scripts don't pick up whether a script successfully completed or not (with added process.exit(0) / process.exit(1)).

The logs always report that job completion wasn't reported at all:


# Job failed at 2023/08/13 14:34:41 (GMT+0).
# Error: Process exited without reporting job completion.
# End of log.

Interestingly a test shell script succeeds:


#!/bin/sh

# Enter your shell script code here
pwd

Am I overlooking something regarding the reporting of the status or might this be related to this issue?

Thank you for the wonderful project!

jhuckaby commented 1 year ago

@Hahlh I cannot help with someone's 3rd party docker compose configuration. I have no idea how that product works. Sorry. Just note that Cronicle is very sensitive to server hostnames, and Docker generates random ones or something. See the troubleshooting guide: https://github.com/jhuckaby/Cronicle/wiki/Troubleshooting#server-hostnames

As for your exit code situation, that is as designed. Cronicle Plugins communicate with JSON over STDIO. See the docs for details: https://github.com/jhuckaby/Cronicle/blob/master/docs/Plugins.md#json-output

The Shell Plugin is designed specifically to interpret shell output codes, and it converts that back to a JSON formatted response for Cronicle.

Hahlh commented 1 year ago

@Hahlh I cannot help with someone's 3rd party docker compose configuration. I have no idea how that product works. Sorry. Just note that Cronicle is very sensitive to server hostnames, and Docker generates random ones or something. See the troubleshooting guide: https://github.com/jhuckaby/Cronicle/wiki/Troubleshooting#server-hostnames

As for your exit code situation, that is as designed. Cronicle Plugins communicate with JSON over STDIO. See the docs for details: https://github.com/jhuckaby/Cronicle/blob/master/docs/Plugins.md#json-output

The Shell Plugin is designed specifically to interpret shell output codes, and it converts that back to a JSON formatted response for Cronicle.

Thanks a lot for the pointer, I did indeed miss that part of the docs.

And no worries, our current setup is functioning well and my main intention was to offer extra information that might reveal a pattern and assist you in debugging the overall issue.