[Feature Request] Scheduled backups

[johnny-tknoska]

Hi there again! As I mentioned before, thank you very much for your incredible work with this tool.

Would it be possible to add the following functionality?

An option that would allow making automatic Backups of the Config, DB, and Devices/Services CSV, in other words, the 3 backups that currently exist, and that could be scheduled for a specific time to execute (logically stopping all scans as I indicated in my previous message).

Thank you!

[leiweibau]

I would hold off from implementing an automatic cleanup of the backups, but make a note of this for the future.

The current implementation is as follows:

• Configuration of the interval in the crontab syntax within pialert.conf
• NO integrity check is performed for the database backups
• The automatic backups use the same naming scheme as the manual backups
• The smallest time interval is 1 minute (but should not be used because it makes no sense). However, it would allow a time like 11:34 a.m. every day.
• If a scan is active at the start of the backup, the script waits until the scan has finished and any database transactions for reporting have been completed. The backup is then created with the help of the sqlite3 client. During the time the script is waiting or performing the backup, the status box on the settings page will say "Waiting" or "Pending"
• The configuration file and the database are currently covered in the automatic backup. CSV is not yet implemented, as I am not sure about this either.

[johnny-tknoska]

Awesome! thanks so much. I really think it's a very nice feature and your implementation exceeds my expectations.

[leiweibau]

I made a "small" mistake. As I configured slightly different cronjobs in my development system, I made an incorrect statement.

The backup task is executed via the cronjob of the "internet_IP" function, so it can only be started every 3 minutes and not every minute. Only times every 3 minutes work here.

*/3  */2 *   * *   - ok
30   1   *   * *   - ok
*/15 4   */2 * *   - ok
10   2   */3 * *   - not ok

[leiweibau]

Update released with https://github.com/leiweibau/Pi.Alert/commit/8f9738669bbc3204d1804ab0cfd507d191ac71f0

[johnny-tknoska]

It works very well. Thank you very much!

Just one comment:

I noticed that when you do a manual backup, the .zip file created is owned by the www-data user, but with the new automatic backup feature, the file is owned by the pialert install user. I don't know if this is important or not for a possible DB recovery.

Also, in case you want to implement it in future releases, I have written a bash script for myself to clean up old backups.

My script looks like this:

• Maintenance: Remove older Pi.Alert backups, maintaining just the copies_to_maintain copies more recent

db_folder="$HOME/pialert/db"

copies_to_maintain=3

files=($(ls -t $dbfolder/pialertdb*.zip 2>/dev/null))

if [ ${#files[@]} -gt $copies_to_maintain ]; then files_to_delete=("${files[@]:$copies_to_maintain}") for file in "${files_to_delete[@]}"; do sudo rm "$files" done else : fi

[leiweibau]

You have recognized this correctly. The manual backup is triggered by the web server, while the automatic backup is created by the user crontab, which leads to the different permissions. A small code adjustment should correct this.

I will certainly have to make one or two tweaks, as such implementations seem small at first, but in the end there are many other things involved.

Thank you for pointing this out. 👍

An automatic backup cleanup will be available in the near future. I just wanted to release this feature and a few other things first

[johnny-tknoska]

Hi! I think there's a bug in the cron interpretation for scheduled backups. Yesterday morning (04/16/2024) I set up the cron expression in Pi.Alert config for scheduled backups as follows: 03 19 /2 *

So backups have to be done at 19:03 on 04/17, 04/19, 04/21, etc., but I noticed that a backup was done at 19:03 on 04/16.

I will check to see if a backup is done on 04/18, because if it is, it's definitely a bug or maybe it's not possible to set cron expressions like this and then I will change it to do weekly backups, something like this: 03 19 0.

Please, let me know. Thanks!

[leiweibau]

Thanks for the message. I thought I could run the crontab check with one function. However, contrary to the minutes, hours and day of the week, there is no 0 for days and months, which is why the job shifts by -1. I will fix that. It also somehow didn't look wrong that the even days were executed for */2.

[leiweibau]

Update released with https://github.com/leiweibau/Pi.Alert/commit/91d0f5770f2498e084d0167b202029f841974e4c

[graphixillusion]

I have set in the pialert.conf this two lines:

AUTO_DB_BACKUP = True AUTO_DB_BACKUP_CRON = '0 '

but no backup is done. Is the only configuration to make it works?

[leiweibau]

AUTO_DB_BACKUP_CRON = '0 '

As you allready know, this means, every hour at Minute 0. Can you show me the "Internet / Speedtest / Updatecheck" log from this moment? You may have to wait a minute for the log to be created. Does the journal show that the cronjob was executed? Are you using an LXC environment for Pi.Alert?

[johnny-tknoska]

AUTO_DB_BACKUP = True AUTO_DB_BACKUP_CRON = '0 '

but no backup is done. Is the only configuration to make it works?

I guess the problem is that this cronjob does not get run at integer multiples of 3 minutes as required.

(# The shortest interval is 3 minutes. All larger intervals must be integer multiples of 3 minutes.)

Let's try for example 3 and I think it will work

[leiweibau]

"0" should actually also work. No multiplication is used for the calculation, instead a list is created with the values 0-59 and these are filtered in steps of 3. The limitation to 3min results from the cronjob that executes this script.

[leiweibau]

Also note that a scan starts every 5 minutes, i.e. also "0", which is waited for before the backup starts. This is intentional and must be the case.

[graphixillusion]

AUTO_DB_BACKUP_CRON = '0 '

As you allready know, this means, every hour at Minute 0. Can you show me the "Internet / Speedtest / Updatecheck" log from this moment? You may have to wait a minute for the log to be created. Does the journal show that the cronjob was executed? Are you using an LXC environment for Pi.Alert?

I'm not running any LXC env, i'm using Pi.Alert in raspbian (pizerow). In the journal there is no trace of the running backup and in the crontab i just have these lines.

0 3 * * 1      python3 $HOME/pialert/back/pialert.py update_vendors >$HOME/pialert/log/pialert.vendors.log  2>&1
*/3 * * * *    python3 $HOME/pialert/back/pialert.py internet_IP    >$HOME/pialert/log/pialert.IP.log       2>&1
*/5 * * * *    python3 $HOME/pialert/back/pialert.py 1              >$HOME/pialert/log/pialert.1.log        2>&1
0 2 * * */4    python3 $HOME/pialert/back/pialert.py cleanup        >$HOME/pialert/log/pialert.cleanup.log  2>&1

This is the log

Check Internet IP
    Timestamp: 2024-04-27 13:21:00

Retrieving Internet IP...
    removed

Retrieving previous IP...
    removed
    No changes to perform

Skipping Dynamic DNS update...

Auto Speedtest...
    Skipping Speedtest... Not activated!

Auto Update-Check...
    Crontab: 0 3,9,15,21 * * *
    Version Check function was NOT executed.

Auto Backup...
    Crontab: 0 * * * *
    Backup function was NOT executed.

DONE!!!

I've removed the ip address.

[leiweibau]

Thank you for your feedback. With your configuration, the log of, for example, 14:00 or 15:00 would be interesting. Nothing happens at 13:21 either 😉

I hope I haven't caused any confusion, but by journal I didn't mean the Linux journal, but the journal in Pi.Alert

[graphixillusion]

Yes, i mean no trace of database backup in the pialert's journal. I just have this kind of entries:

2024-04-27    15:00:00 | Update Check | cronjob | No newer version found

Maybe i must manually add something to the crontab?

[leiweibau]

Maybe i must manually add something to the crontab?

No, that is not necessary.

1. Is this a new installation or an update?
2. Is it possible to create a manual backup via the web interface?
3. Please change AUTO_DB_BACKUP_CRON to '/3 ' (every 3 min) so that the waiting time to get a result is not so long.
4. Did you saved the "Internet / Speedtest / Updatecheck" log from "2024-04-27 15:00:00", and can you show it to me? The difficulty lies in the fact that this log only appears approx. 30 seconds after the scans have been completed, because the script waits until they are finished.
5. If you go to the console/terminal of the Pi.Alert host and enter the command "date", does the date and time match your current time?
6. Does the directory "pialert/db/temp" exist and if so, what permissions does it have? What permissions does "pialert/db" have?
7. Please execute this command in the directory "pialert/db/" sqlite3 pialert.db .backup ./temp/pialert.db. Does an error message appear?

[graphixillusion]

1. update
2. yes, manual backup from web interface is possible and it's working
3. done
4. sadly nope i didn't see the log at 15.00
5. date matches
6. the temp folder exist and have the permission drwxrwxr-x user:www-data
7. this error appear: missing FILENAME argument on .backup

Now in the log i have

Auto Backup... Crontab: /3 * Backup function pending.

but no backup is done

[leiweibau]

Sorry, i gave you the wrong command: sqlite3 pialert.db ".backup 'temp/pialert-test.db'"

Here is a brief "simplified" explanation of how the backup function works.

The backup is started via the cronjob */3 * * * * * python3 $HOME/pialert/back/pialert.py internet_IP. If "AUTO_DB_BACKUP" is set to True, the backup function is started, which also checks whether the time for the backup has been reached. While this function is running, it creates a file with the name ".backup" in the "pialert/back" directory. Regardless of whether a backup is created or not, this file is deleted again at the end of the function. In the case where the backup is to be created, but the scan is still running, the script waits and does not yet delete this file. The script recognizes whether a scan is running by the fact that it finds a ".scanning" file in the "pialert/back" directory, which should be deleted at the end of the scan. If the 3 minute cronjob is now started again while the script from the cronjob 3 minutes earlier is still waiting, the new cronjob now finds the .backup file and outputs the text "Backup function pending".

At this point I would now do the following.

1. via Settings "Toggle Pi.Alert on/off" set the timer to e.g. 24h and switch off Pi.Alert.
2. switch to the terminal and open the "pialert/back" directory
3. execute ls -al. The two files .backup and .scanning may now appear here. Delete these two files.
4. Now either restart or sudo killall python3.
5. start Pi.Alert via Settings "Toggle Pi.Alert on/off".

[graphixillusion]

sqlite3 pialert.db ".backup 'temp/pialert-test.db'" <--- this command doesn't give any error.

I did some test and now i can say that the backup is performed only when:

1. I stop Pi.Alert with the "Toggle Pi.Alert on/off" via Settings.
2. I remove .backup and .scanning in the pialert/back directory
3. I restart Pi.Alert with the "Toggle Pi.Alert on/off" via Settings.

Like this it will creates a backup a little while after the restart and then it stops again. It's something like .backup and .scanning files are always present.

Infact, in the status page i always have:

"scans currently running: (next scan in: xx:xx) and Auto-Backup Enabled / Pending

permissions of both .scanning and .backup files = -rw-r--r-- user:user

[leiweibau]

Since it executes the function once, the script can obviously also delete the ".scanning" file. I would therefore rule out a problem with the permissions, at least for the time being. What is the creation date of the backup?

It is also possible that my instruction to set the backup time to '/3 ' has only now made things really bad. For example, take this configuration '0 12 *', stop Pi.Alert and delete the two files and see if it works now.

[graphixillusion]

I did this modification but still doesn't work.

[leiweibau]

I admit that I'm a bit at a loss right now. If I was able to recreate your setup, I'm sure I could find out more, but unfortunately I don't have a Pi Zero.

Questions I still have would be:

• How long does a scan usually take for you? The Pi Zero is not necessarily the fastest, precisely because it only has a WLAN interface.
• Do you use website monitoring and/or ICMP monitoring? If so, with how many sites and/or hosts?
• How large are the networks/network? (/24, /16 or several networks)

[graphixillusion]

I have just the arp-scan enabled in a /24 subnet, 9 devices in total in the network. The pialert.conf is pretty much the default one: the only modification i did is set --localnet --interface wlan0 to the arp-scan settings. No website monitoring or ICMP. The strange thing is that according to the status page the scan never stop in my case: i always have "scanning currently running" and the timer next to it saying next scan in ... When the timer stops the counter restarts from the beginning.

So i think a workaround would be a cron script that starts and temp disable the Pi.Alert scan, launch the backup and then restart the Pi.Alert scan. Which is the command that Pi.Alert launches for the autobackup job?

[leiweibau]

Which is the command that Pi.Alert launches for the autobackup job?

Basically it is this sqlite3 pialert.db ".backup 'temp/pialert.db'". This database is then compressed and given a file name containing the current date. If you have not done this before, the "pialert/db/temp" directory should also contain the pialert-test.db from before. You may have to complete the paths, depending on the "location" from which you want to run the script. E.g.: /usr/bin/sqlite3 /home/user/pialert/db/pialert.db ".backup '/home/user/pialert/db/temp/pialert.db'".

Please remember to set the auto-backup to False. Too bad I wasn't able to solve your problem

timer next to it saying next scan in ...

Ignore the timer. This is really just a time reference and has no relation to the actual scan.

I suspect that the scans are building up and therefore the backup cannot be created. To test this, you could, as a last attempt, open "pialert.py" in the "pialert/back" directory and in this line,

https://github.com/leiweibau/Pi.Alert/blob/59e4a8715de2ba26c31a77ea8d8f37bf31fcdd12/back/pialert.py#L862

Set the value for "retry" to e.g. 3. This reduces the duration of the arp scans.

[graphixillusion]

Sadly still nothing...

[leiweibau]

That is a pity. I'm really at a loss at the moment.

[graphixillusion]

For now i'm using a bash script with a crontab which runs every day at 6 am. The script stop the scan, make the backup and start the scan again.

[graphixillusion]

Since i'm using the backup scripts i haven't notification's badges about devices anymore. The only notifications i got are Pi.Alert is paused/Pi.Alert is reactivated. I didn't change anything else, just this one: https://github.com/leiweibau/Pi.Alert/issues/259#issuecomment-2081597208

Any clue?

[leiweibau]

You could run ./pialert-cli set_permissions in the pialert/back directory.

Another possibility would be to simply run the update script again. This will reset the changes to the pialert.py script.

[graphixillusion]

I've run the set_permissions command but still nothing, no notification about device's connect/disconnect

[graphixillusion]

I would like to report that i went super clean: i reinstalled everything from scratch including the OS. Everything's working now as it should.

[leiweibau]

I'm very happy to read that. Thanks for the feedback