search

jhuckaby / Cronicle

A simple, distributed task scheduler and runner with a web based UI.
http://cronicle.net
Other
3.94k stars 391 forks source link

Daily cron for Cronicle data export giving out errors #779

Closed 1992kk closed 3 months ago

1992kk commented 5 months ago

Summary

Following the steps to export daily backup of cronicle data, we've setup a cron in Cronicle itself to run on the master node. When script is executed manually, it works well. If the cron is run manually, it works well. However, when the cronicle server executes its scheduled run, it fails.

Steps to reproduce the problem

  1. Create a bash script
  2. Create an event in cronicle to run this bash script. Script contents can either be pasted directly in the UI or wrapped in a file and called.
  3. Once the cron is executed as per schedule, error is observed

Your Setup

Operating system and version?

Rocky linux 8.7

Node.js version?

v16.19.1

Cronicle software version?

0.9.53

Are you using a multi-server setup, or just a single server?

1 master, 3 workers

Are you using the filesystem as back-end storage, or S3/Couchbase?

filesystem

Can you reproduce the crash consistently?

Yes

Log Excerpts

`node:internal/validators:162 throw new ERR_INVALID_ARG_TYPE(name, 'string', value); ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined at new NodeError (node:internal/errors:387:5) at validateString (node:internal/validators:162:11) at Object.dirname (node:path:1276:5) at /z/cronicle/node_modules/pixl-server-storage/transaction.js:318:44 at FSReqCallback.oncomplete (node:fs:192:23) { code: 'ERR_INVALID_ARG_TYPE' } ERROR: Parameter problem: Nothing to upload.

Job completed successfully at 2024/06/24 14:00:00 (GMT+0).

End of log.`

1992kk commented 5 months ago

Not sure but changing the installation directory to /opt/cronicle fixes this issue.

Ref : https://github.com/jhuckaby/Cronicle/issues/778

jhuckaby commented 5 months ago

Yeah, Cronicle really wants to live in /opt/cronicle. There are a bunch of things that expect that base path, particularly the control.sh script and all the CLI commands.

I'll add a louder mention of this in the docs.

strefethen commented 3 months ago

@jhuckaby curious if there any way to modify control.sh to work with an install at /opt/bitnami/apps/cronicle?

I ask because I was trying to run control.sh maint and got the above error.

Unfortunately, I originally setup Cronicle using the above path from the beginning (as root) now I'm running into a problem and the server won't start and I'm trying to debug it without much luck. I tried to run the control.sh export command but it fails with the above error and I'm getting a sinking feeling. I have a snapshot of this instance but it's from June. I have a lot of jobs on this server and I'm unable to export a txt backup. Of course, I realize now I probably should have been running this all along.

I see evidence of job shell scripts in the data directories but I can't make much sense of the directory structure there.

I was able to run ./bin/storage-repair.js and an error was found/corrected.

But attempting to restart the server I see the following appear very quickly (not normal):

./bin/control.sh start: Starting up Cronicle Daemon...
./bin/control.sh start: Cronicle Daemon started

...and the server isn't started.

The logs/crash.log shows:

[1723832161.997][2024-08-16 11:16:01][ip-172-26-2-105][7545][crash][debug][1][Emergency shutdown: Could not rollback transaction: logs/events/e
lse7dln39i: Failed to restore record: logs/events/else7dln39i/-176: Failed to rename file: logs/events/else7dln39i/-176: data/_temp/f8559408407
94774a5a2affebff5aa2e.json.tmp.6030912: EACCES: permission denied, rename 'data/_temp/f855940840794774a5a2affebff5aa2e.json.tmp.6030912' -> 'da
ta/logs/f8/55/94/f855940840794774a5a2affebff5aa2e.json'][]

/logs/Cronicle.sh shows:

[1724085156.788][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][2][Cronicle v0.9.20 Starting Up][{"pid":5154,"ppid":1,"node"
:"v16.18.1","arch":"x64","platform":"linux","argv":["/opt/bitnami/node/bin/node","/opt/bitnami/apps/cronicle/lib/main.js"],"execArgv":[]}]
[1724085156.792][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][1][WARNING: An old PID File was found: logs/cronicled.pid: 5
076][]
[1724085156.792][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][2][Old process 5076 is apparently dead, so the PID file will
 be replaced: logs/cronicled.pid][]
[1724085156.792][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][9][Writing PID File: logs/cronicled.pid: 5154][]
[1724085156.793][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][9][Confirmed PID File contents: logs/cronicled.pid: 5154][]
[1724085156.794][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][2][Server IP: 172.26.2.105, Daemon PID: 5154][]
[1724085156.795][2024-08-19 09:32:36][ip-172-26-2-105][5154][Cronicle][debug][3][Starting component: Storage][]

I see a lot of files in ./data/_temp and an error in Filesystem.log:

[1723832161.996][2024-08-16 11:16:01][ip-172-26-2-105][7545][Filesystem][error][file][Failed to rename file: logs/events/else7dln39i/-176:"queue":0,"chain":"","chain_error":"","notify_success":"","notify_fail":"","web_hook"
 data/_temp/f855940840794774a5a2affebff5aa2e.json.tmp.6030912: EACCES: permission denied, rename 'data/_temp/f855940840794774a5a2affebff5ay_sustain":0,"log_max_size":0,"notes":"","now":1721994420,"id":"jlz2n0mumui","time_st
a2e.json.tmp.6030912' -> 'data/logs/f8/55/94/f855940840794774a5a2affebff5aa2e.json'][]

However, I don't see a folder with the name logs/events as mentioned above.

Lastly, I see this in recovery.log which references this same file.

[1724086263.711][2024-08-19 09:51:03][ip-172-26-2-105][5474][Storage][error][fatal][Fatal transaction error: Could not rollback transactio
n: logs/events/else7dln39i: Failed to restore record: logs/events/else7dln39i/-176: Failed to rename file: logs/events/else7dln39i/-176: d
ata/_temp/f855940840794774a5a2affebff5aa2e.json.tmp.2: EACCES: permission denied, rename 'data/_temp/f855940840794774a5a2affebff5aa2e.json
.tmp.2' -> 'data/logs/f8/55/94/f855940840794774a5a2affebff5aa2e.json'][]
[1724086263.712][2024-08-19 09:51:03][ip-172-26-2-105][5474][crash][debug][1][Emergency shutdown: Could not rollback transaction: logs/eve
nts/else7dln39i: Failed to restore record: logs/events/else7dln39i/-176: Failed to rename file: logs/events/else7dln39i/-176: data/_temp/f
855940840794774a5a2affebff5aa2e.json.tmp.2: EACCES: permission denied, rename 'data/_temp/f855940840794774a5a2affebff5aa2e.json.tmp.2' -> 
'data/logs/f8/55/94/f855940840794774a5a2affebff5aa2e.json'][]
[1724086263.712][2024-08-19 09:51:03][ip-172-26-2-105][5474][Storage][debug][1][Exiting][]

Apologies, I'm sort of grasping at straws trying to get things going again.

Any tips/thoughts would be greatly appreciated.

jhuckaby commented 3 months ago

@strefethen It sounds like you need some professional-level hands-on support, but this is just an open-source project with a single author. I just don't have the time to troubleshoot your installation.

In general, I can tell you this: Cronicle REALLY needs to live in /opt/cronicle. It won't be happy anywhere else. Can you just do a fresh install there, and then copy over your entire data directory?

This is the real core issue you need to address:

EACCES: permission denied, rename 'data/_temp/f855940840794774a5a2affebff5aa2e.json
.tmp.2' -> 'data/logs/f8/55/94/f855940840794774a5a2affebff5aa2e.json

Are you running these commands as root? EACCES means that whatever user is running Cronicle or the scripts does not have write access to the filesystem.

I'm so sorry, but I just don't have the time to offer professional support services. This is s 100% free open source project, and I am doing this in my spare time.

Apologies, but good luck.

strefethen commented 3 months ago

Totally understand. This setup has worked perfectly until this point. I greatly appreciate you reading my post above and offering a pointer.

Btw, my need for "higher" level help aside, I'd be happy to sponsor Cronicle monthly, which is something I already do with Plausible.io and other OS projects.

image

https://github.com/sponsors

strefethen commented 3 months ago

An update...

I cleaned up the jobs folder as per your comment here:

https://github.com/jhuckaby/Cronicle/issues/304

I then did:

sudo chown -R bitnami:bitnami data/
sudo chmod -R 755 data/

And the server is now back up. Not sure why this problem started, seemingly, all of the sudden.

Thanks again for your help.

I'll work on getting a new install under /opt/cronicle, it's overdue.

jhuckaby commented 3 months ago

I'm glad you were able to figure it out. Thank you for the offer to sponsor Cronicle! I really appreciate that. I will look into Plausible when I have some time.

Cheers!