logging: errors should go to STDERR

[IzzySoft]

Currently, there's no -q parameter to make scan "quiet" e.g. when running from Cron. Redirecting STDOUT to /dev/null redirects ALL output, including errors – so one would not notice if automated runs had errors.

Errors should hence be directed to STDERR, not STDOUT.

A -q parameter would be welcome, too :wink:

[IzzySoft]

Speaking about logging: would be great to

• have the log level configurable
• use a different log level when a real update is done; maybe introduce a NOTICE level between INFO and WARNING for that? Logging levels I know from different places/logging-libraries, ordered by importance: EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG (your library only knows about error, warning, info, debug, i.e. 4 out of the 7)
• logging messages from ERROR and up to STDERR instead of STDOUT (see above)

[SanmerDev]

use a different log level when a real update is done

What does it mean to the real update? 🤔

[IzzySoft]

That you can better control the output. Currently, it's a bit confusing (e.g. why does removing the oldest ZIP as configured yield a WARNING? That's expected behavior and should be INFO at best) and hard to read if you just want to a) see if there were any updates, b) just want a mail from cron in case something failed, or c) want to automatically evaluate if there were any updates at all and a sync should be triggered.

I had to define myself an alias to keep things under control, calling sync like

util/cli.py sync |egrep -v '(update module from|: already the latest|repo_name:|repo_url:|max_num:|show_log:|log_dir:|number of modules|unlimited-hotspot)'

• unlimited-hotspot yields an error on each run as its Github repo disappeared. I still want to keep it, but cannot exclude it from the update run as there's no option for that (yet), see #13 & #14
• the other details are not wanted when running from cron, else one gets a long mail on each run. I still want mails when something goes wrong, but cannot > /dev/null the output as ERRORs are not properly sent to STDERR and thus would go to /dev/null as well
• being able to define what should be logged to screen and what to the log file allows for only showing what's needed (e.g. log INFO and up to the log file, but only NOTICE and up to screen¹ – so you see all you need on a manual run, can "quiet" the screen output for cron runs by redirecting STDOUT to /dev/null – and on errors can always investigate the log file.

To the update itself it doesn't mean a thing, as nothing would change there. It's just the logging itself we're talking about here.

---

¹ assuming that e.g. INFO would be mostly what it is now except for when there are really changes – though what goes to which level might be a different discussion later. Using my filters from above, I'd e.g. send

• update module from|: already the latest|repo_name:|repo_url:|max_num:|show_log:|log_dir: to DEBUG
• when a module was updated, or a ZIP pruned: INFO
• when a module could not be updated: WARNING
• ERROR when something goes wrong
• CRITICAL or ALERT when the index would break (in that case, the old one should be retained)

Just rough brain-storming, not having all cases present :wink:

[SanmerDev]

This will take some time, currently I am refactoring and optimizing sync😶‍🌫️.

[IzzySoft]

Yes, sure – you make the schedule as you see fit of course! I saw you've already pushed a bunch of changes :+1: Please let me know when any of those require different handling when updating to HEAD on my end.

[SanmerDev]

Currently, my advice is not to follow HEAD, because sync has a lot of places that will be changed, so I can't ensure the stability of util, I will release it when everything is OK.

I plan to implement version control to ensure that every version of MRepo has the util available. Also, this should make it easier for repositories to migrate when JSON structure changed.

[IzzySoft]

Thanks! Then I wait for your signal on when an update should be safe (together with the details to change or to watch out for). If there's a "stable point" on the road to the next release which you want me to test out, you can give that signal as well (always a good idea having another pair of eyes checking things with a possibly different setup, so the release is as "clean" as possible :wink:)