Too many cronjobs

[thgh]

I suggest to remove all crons and add just one: /opt/minepeon/etc/cron.d/croncontrol.php

That php script will load settings.inc.php and then these modules in this order:

• mmaUpdate
• rrdUpdate (graphs)
• alert (save alert status in a file)
• mail (based on alert module)
• donation (only overwrites miner.conf, recover will do the restart)
• recover (based on alert module and donation module)
• mmaCheck

(donation is also possible without restart)

[ghost]

It's hard to just add one cron job when some need to update xx amount of seconds, xx amount of minutes, xx amount of days.

It's typical for linux systems to have lots of cron jobs.

In the case of MobileMinerApp, you have to update the server with stats every 20sec to 1min 35sec, as it will become inactive after 2 minutes and show said device offline. I'm not sure the limitations on receiving commands, hence why I put the cron on a 2 minute timer there.

I could see the alert as a 15min cron or 30min cron.

Also, debugging reasons another reason you want to separate the cron jobs, where as one cron job could fail, the others won't. Grouping them into one job means, if one fails, which one, when, under what circumstances, etc...

It's really not good practice to group cron jobs under a single file. You can have cron job groups, which are technically single files in a folder and cron runs each group on a timer, usually that's when you have multiple cron jobs executing on the same time (ie: 4 jobs every 15 min).

[MineForeman]

I get where your coming from Thomas, every time a cron job loads at a minimum it then needs to load settings.inc.php and then the conf file. Both of these are both CPU and disk hits.

I am also a strong believer in linux standards, cron and run-parts, it means when a developer (there are three of us now) needs to create an additional job all he/she needs to do is drop a executable php file into place in the correct location.

I am even thinking of going to something that Thomas will probably hate me for :P . (and MobileMiner would need to fit into)

Something like;-

run-parts

1 * * * * run-parts /opt/minepeon/etc/cron.d/1min /2 * * * * run-parts /opt/minepeon/etc/cron.d/2min /5 * * * * run-parts /opt/minepeon/etc/cron.d/5min /30 * * * * run-parts /opt/minepeon/etc/cron.d/30min 0 * * * \ run-parts /opt/minepeon/etc/cron.d/hourly

It add's more CPU and disk.... But it also adds a more maintainable code base.

The other option is to almost rewrite cron into a php script that gets called every minute that then decides what to run like Thomas describes that then decides what additional scripts to call by loading settings.inc.php (and then the conf file).

The reduced code complexity reasons lean me towards option one (run-parts) and efficiency leads me towards croncontrol.php .

I would like to hear both of your guys thoughts on my throughs, both have their advantages and disadvantages.

Neil

[ghost]

The run-parts is what I meant by 'corn job groups', so I agree with that.

I'm completely in understanding of the CPU/Disk hit, but if you want to provide things to people in near real-time, it's needed, plus, being honest, it's really no more of a hit than viewing a the site. There is probably more logging going on (ie: more disk writes) when viewing the site than there are running the crons (unless you have all logging turned off for http, Neil, I honestly haven't checked).

[thgh]

I stand by my point. Except for donation, I see all those cronjobs as 1min crons.

I also would like to bring this to the table:

• Updatability & compatibility: easily add modules, easily change execution interval
• Stability: currently some of the code interferes (restart & stats & donation)

@hwilliams-directnic People only view the website a couple of minutes a day. That's no cpu/disk issue.

I have a stub of croncontrol.php and rewritten donation code that donates without restart. Another killer feature I would say :-)

[ghost]

Adding a config (or appending to the current config) to allow users to select/deselect modules they want added/running in a settings tab, could also have it check to see if said file(s) for module exists, if it doesn't download a tar.bz file from the server on the internet.

PHP exec a wget to a PHP page that's connected to a SQL database where we just pass in the module name with the url call and it will lookup the current stable version of the module, redirect to the download. This method could also be used for keeping the system up to date, once downloaded extract, put files in respected places (can also have it append needed settings to the settings file), then delete the temporary downloaded tar.bz file.

And I'm not so sure people view the website a couple times a day, I have a monitor open with mine running almost 80% of the day (as I like to monitor things). Come monday I'll have four blades here, so I'll be playing with that a lot as well.

But having an system upgrade cron on a minute interval, that's not really conventional. Even having it every couple of hours isn't conventional, every X amount of days would lean more towards ideal. However then you do run into the whole RTC issue if the machine continues rebooting. So it might have to be every X amount of hour(s), I still would advice against it being minutely.

[thgh]

That config for modules is already in the repo. Each module has its own namespace and an xxxxEnable option.

Currently I plan to keep all modules inside the repo, so everyone has them and does not need any root rights to enable them. @hwilliams-directnic feel free to build an extended module system :-)

I hoped the update module would be a simple git pull. I'm not sure about automatic updates, as some say: don't fix what's not broken. Do you know any exploits on pulling from git?

For me, the webui using significant resources is no factor in the discussion on how efficient the base system should be. Neil communicated that efficiency was part of the vision and I couldn't agree more.

I hope many people get bored soon looking at the colorless graphs. That's one reason why I brought back the static graphs as default. Btw, except for the access log, there should be no disk writes when refreshing.

[thgh]

@MineForeman I'm ok with both ways, but in the multi cronjob way, I hope someone rewrites the cron code clean and efficient without too much sleep(..)

[MineForeman]

Sorry, my real job got in the way of the discussion.

However I did set up some tests before I got called into the meeting and I got some interesting results;-

1st call of settings.inc.php from either user space or apache fully loads both it and the config files into memory, from that point on it is no longer a disk hit. The filesystem even tracks the file for changes so the disk is never hit again unless the files change underneath they never get reloaded.

Even the sleep() calls are not much of an issue because they are run in user space and not kernel or even apache. Even as it stands the cron jobs are a minute load on the system. (Having said that, looking at the cron jobs I wrote I do need to do a better job).

So, at the moment unless anyone has any great objections I am going to implement the 1/2/5/15/30/60 run-parts system mentioned above to make it easy for 'add-on' developers just to pop their files in place with a tar -xzf and have there add-on just to pop into place and start working (almost, I mostly take my inspiration for this because that is how the wordpress add-on system works, and it works very well).

It does not address as to how to start to add the settings for these add-ons to the WebUI. I think that is the problem that we should really start tackling.

Neil

[thgh]

OK, don't forget to make one noon cronjob for donations :-)

Solution for settings: It's possible to load a list of partials (combination of ngRepeat with ngInclude) Only problem is that addon writer has to know how to write an angular form, but this can be solved with a settingspartial generator

[ghost]

I assume keeping the settings in JSON, it would be extremely easy just to pass the settings from the form, to a PHP script, have it compare the current settings, change/add what's needed, then rewrite the file or append to the file.

Say you wanted to turn on MMA Module, you click the box (and enter settings if needed) then save, the backend would just post to a script, compare the current configuration, then recreate or append the file.

With that being said if the module was disabled, then the cron checking the module would never actually run it, but then if it was enabled, it would get picked up from the config and ran the next time the cron runs.