Improvements to Dropbox and Google Drive backups

[877dev]

Hi,

I've been working over the weekend and this is working fine for me now. Let me know what you think.

Increased local backups from 5 to 7 Fixed local files not being deleted : permission denied Improvements to Dropbox - now last 7 files are kept and the rest deleted

[877dev]

19-11-2019 rclone now syncs with local backup files (most recent 7 backups are kept) Tidied up comments/echo statement.

[gcgarner]

can you put the delete portion of dropbox inside this

the grep -c counts if there are any strings containing "backup" if not then don't delete

    if [ $( echo "$files" | grep -c "backup") -ne 0 ] ; then
              # delete code here
    fi

This will avoid the error if there are less than 7 files and there is nothing to delete

[877dev]

Great suggestion, I learned how the grep command works. I've made the changes...

[stevenveenma]

Hi, I store the backup files on my NAS. I succeeded in mounting and automating the backup. Perhaps a short introduction on this would be helpful for people that want to use their NAS instead of a cloud?

[gcgarner]

Could you maybe give a sample code of how you did it (minus any sensitive info)

Scp/smb/ftp or whatever you use

On Fri, 22 Nov 2019, 13:36 Steven Veenma, notifications@github.com wrote:

Hi, I store the backup files on my NAS. I succeeded in mounting and automating the backup. Perhaps a short introduction on this would be helpful for people that want to use their NAS instead of a cloud?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gcgarner/IOTstack/pull/78?email_source=notifications&email_token=ALECSYNAT6IBZW6D5AJDWHDQU672BA5CNFSM4JOKBKJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE5MHDI#issuecomment-557499277, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALECSYNU2VDXAHFMBKKPUFDQU672BANCNFSM4JOKBKJA .

[stevenveenma]

Step 1: Install the CIFS Utils pkg sudo apt-get install cifs-utils

Step 2: Create a mount point sudo mkdir /mnt/local_share

Step 3: Mount the volume sudo mount -t cifs //<vpsa_ip_address>/<export_share> /mnt/<local_share>

In my case I needed to add my credentials and the version (because my NAS is quite old) sudo mount -t cifs //<vpsa_ip_address>/<export_share> -o user=<username>,passsword=<password>,vers=1.0 /mnt/<local_share>

Step 4: Startup Crontab and make a rule that copies the docker user-data to the NAS sudo crontab -e * 3 * * * sudo cp -avr /home/pi/IOTstack/volumes /mnt/<local_share> This makes a daily copy at 3 o'clock of the data generated by the Docker containers. Old files are replaced by new versions, new files are added, Removed files will not be removed on the NAS.

This works fine but the mount point has to be set manually after a reboot. I tried to automate this using fstab. But unfortunately this failed despite various tried suggestions amongs delays and syntax options I found in fora. I supsect the docker network that is build up after the reboot. For the ones who want to reproduce this add to fstab: //<vpsa_ip_address>/<export_share> /mnt/<local_share> -o username=<username>,passsword=<password>,vers=1.0 0 0

[gcgarner]

@stevenveenma

I'm going to have to think of a clever way to add this into the menu structure.

I think that adding the share into the fstab would be a better option. Better yet do the fstab to the backups folder and you are done

[stevenveenma]

@stevenveenma

I'm going to have to think of a clever way to add this into the menu structure.

I think that adding the share into the fstab would be a better option. Better yet do the fstab to the > backups folder and you are done

I agree that adding the share in fstab is preferable, but I haven't succeeded to get this to work. I reproduced this behaviour on another RPI without Docker, so this is not the reason. If you succeed on this I would gladly hear it.

[ristomatti]

@stevenveenma It might be that network connectivity is not up yet at the time fstab is read. I've been using autofs successfully also on a RPi. It works so that the network drive is mounted on the fly when the mount dir is accessed. I'm now on mobile but I can paste example configs in case someone is interested. Not sure if this PR is the right place to discuss this though?

[gcgarner]

I think i found the issue with the errors

pi@raspberrypi4:~/IOTstack $ ~/Dropbox-Uploader/dropbox_uploader.sh list /IOTstackBU
 > Listing "/IOTstackBU"... DONE
 [F] 0         backup - Copy (3).txt
 [F] 0         backup - Copy (4).txt
 [F] 0         backup - Copy (5).txt
 [F] 0         backup - Copy.txt
 [F] 0         backup - Copy (2).txt
 [F] 0         backup - Copy (6).txt
 [F] 0         backup - Copy (7).txt
 [F] 64669887  backup-2019-11-20_1249.tar.gz
 [F] 66135441  backup-2019-11-20_1616.tar.gz
 [F] 157732287 backup-2019-11-27_1754.tar.gz
 [F] 68159197  backup-2019-11-28_1005.tar.gz
 [F] 75263684  backup-2019-12-02_2300.tar.gz
 [F] 77668244  backup-2019-12-09_2300.tar.gz
 [F] 167248915 backup-2019-12-12_0818.tar.gz
pi@raspberrypi4:~/IOTstack $ ~/Dropbox-Uploader/dropbox_uploader.sh list /IOTstackBU | awk {' print $3 '}
"/IOTstackBU"...
backup
backup
backup
backup
backup
backup
backup
backup-2019-11-20_1249.tar.gz
backup-2019-11-20_1616.tar.gz
backup-2019-11-27_1754.tar.gz
backup-2019-11-28_1005.tar.gz
backup-2019-12-02_2300.tar.gz
backup-2019-12-09_2300.tar.gz
backup-2019-12-12_0818.tar.gz
pi@raspberrypi4:~/IOTstack $ 

For whatever reason i have a "backup - copy.txt" and the awk is not catching the spaces correctly causing the errors

trying to see if i can get it parse the data correctly

[gcgarner]

@ristomatti got ahead. This PR has kind of turned into a general improvement on the backups

[stevenveenma]

@ristomatti got ahead. This PR has kind of turned into a general improvement on the backups @ristomatti Sure, I like to hear your suggestion too. I've already spend much time on this issue.

[ristomatti]

@gcgarner @stevenveenma Sorry I didn't find time to reply earlier but in case it's of any use. Setting up mounting with autofs is surprisingly easy. In this example I have an NFS share I'm mounting to but it should work similarly for SMB also.

Install autofs

$ sudo apt install autofs

Create /etc/auto.nfs listing the shared dirs to mount (similar syntax as for fstab)

backup  192.168.1.100:/export/backup

Create /etc/auto.master.d/nfs.autofs to tell where to mount the dirs listed in auto.nfs

/nfs    /etc/auto.nfs

After setting this up, I can just cd /nfs/backup and the contents from 192.168.1.100:/export/backup will be mounted there on the fly if not yet mounted.

For mounting SMB shares autofs seems to have some mechanism for auto detecting shares on the same network but I have not tried this myself.

[gcgarner]

Thanks for getting back to me. The festive season has been keeping me busy.

I'll take a crack at testing autofs when i get the chance.

On Sat, 21 Dec 2019 at 16:07, Ristomatti Airo notifications@github.com wrote:

@gcgarner https://github.com/gcgarner @stevenveenma https://github.com/stevenveenma Sorry I didn't find time to reply earlier but in case it's of any use. Setting up mounting with autofs is surprisingly easy. In this example I have an NFS share I'm mounting to but it should work similarly for SMB also.

Install autofs

$ sudo apt install autofs

Create /etc/auto.nfs listing the shared dirs to mount (similar syntax as for fstab)

backup 192.168.1.100:/export/backup

Create /etc/auto.master.d/nfs.autofs to tell where to mount the dirs listed in auto.nfs

/nfs /etc/auto.nfs

After setting this up, I can just cd /nfs/backup and the contents from 192.168.1.100:/export/backup will be mounted there on the fly if not yet mounted.

For mounting SMB shares autofs seems to have some mechanism for auto detecting shares on the same network but I have not tried this myself.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gcgarner/IOTstack/pull/78?email_source=notifications&email_token=ALECSYNRDEF367ROI3HLWHLQZYPIPA5CNFSM4JOKBKJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHO4V6Y#issuecomment-568183547, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALECSYI3Q5WINHEKUK5GTILQZYPIPANCNFSM4JOKBKJA .

[stevenveenma]

@gcgarner @stevenveenma Sorry I didn't find time to reply earlier but in case it's of any use. Setting up mounting with autofs is surprisingly easy. In this example I have an NFS share I'm mounting to but it should work similarly for SMB also.

Install autofs

$ sudo apt install autofs

Create /etc/auto.nfs listing the shared dirs to mount (similar syntax as for fstab)

backup  192.168.1.100:/export/backup

Create /etc/auto.master.d/nfs.autofs to tell where to mount the dirs listed in auto.nfs

/nfs    /etc/auto.nfs

After setting this up, I can just cd /nfs/backup and the contents from 192.168.1.100:/export/backup will be mounted there on the fly if not yet mounted.

For mounting SMB shares autofs seems to have some mechanism for auto detecting shares on the same network but I have not tried this myself.

Thanks for your input. I tested this but unfortunately this doesn't work either. No mount, no folders visible and no error messages. Comparable to fstab behaviour. Perhaps there is something in my network that obstructs automating this. I think I have to live with the cifs solution.