Dropbox upload not working

[ChirpyTurnip]

Hi,

I'm trying to get the DropBox upload working for my IOTStack backups. The good news is that DropBox part is working fine though it was really hard to set up - I use puTTY and the keyboard inputs (e.g. for the dropbox credentials) don't echo in the console so I'm working blind - that was a major pain! Anyway, I got it to work in the end and if I run:

./dropbox_uploader.sh upload ~/IOTstack/backups/backup/* /

Then I get this:

 > Uploading "/home/pi/IOTstack/backups/backup/backup_2022-01-30_1518.tar.gz" to "/backup_2022-01-30_1518.tar.gz"... DONE
 > Uploading "/home/pi/IOTstack/backups/backup/backup_2022-04-29_1143.tar.gz" to "/backup_2022-04-29_1143.tar.gz"... DONE
 > Uploading "/home/pi/IOTstack/backups/backup/backup_2022-05-29_0724.tar.gz" to "/backup_2022-05-29_0724.tar.gz"... DONE
 > Uploading "/home/pi/IOTstack/backups/backup/backup_2022-05-29_0952.tar.gz" to "/backup_2022-05-29_0952.tar.gz"... DONE
 > Uploading "/home/pi/IOTstack/backups/backup/backup_2022-05-29_0955.tar.gz" to "/backup_2022-05-29_0955.tar.gz"... DONE
 > Uploading "/home/pi/IOTstack/backups/backup/backup_2022-05-29_1003.tar.gz" to "/backup_2022-05-29_1003.tar.gz"... DONE

And the files show up fine in DropBox. So this part is solid.

However, when I run the backup from the IOTStack menu it creates the backup but it doesn't execute the upload. The output I get is this:

### IOTstack backup generator log ###
Started At: 2022-05-29T10-29-17
Current Directory: /home/pi/IOTstack
Backup Type: 3
Backup File: ./backups/backup/backup_2022-05-29_1029.tar.gz
Rolling File: ./backups/rolling/backup_7.tar.gz

Executing prebackup scripts
Some networks were defined but are not used by any service: nextcloud
Service "zigbee2mqtt" uses an undefined network "iotstack_nw"
Rolling Overwritten: True
Backup Size (bytes): 10709300

Executing postbackup scripts
Some networks were defined but are not used by any service: nextcloud
Service "zigbee2mqtt" uses an undefined network "iotstack_nw"

Finished At: 2022-05-29T10-29-26

Items backed up:

./services/
./volumes/
./docker-compose.yml

Items Excluded:
 - No items

### End of log ###

Backup completed.
Press [Up] or [Down] arrow key to show the menu if it has scrolled too far.

There is no upload or even any attempt top upload the files to Dropbox. I dug around and found that there was a need for a 'dropbox' file in the ~/IOTStack/backups folder so I created that. It made no difference. I poked around a bit and found the post_backup_complete.sh file that calls a post_backup.sh file (if one exists) - so I created a post_backup.sh file in the same location and added the command (from above) that runs the upload to Dropbox.

When I test this by just running the post_backup_complete.sh it runs fine and uploads my files as expected. You'd think at that point you've cracked it, but no as this is not executed when I use the backup command from the IOTStack menu.

So now I try to simply run the backup.sh script from the scripts dir and I get this:

### IOTstack backup generator log ###
Started At: 2022-05-29T10-10-59
Current Directory: /home/pi/IOTstack/scripts
Backup Type: 3
Backup File: ./backups/backup/backup_2022-05-29_1010.tar.gz
Rolling File: ./backups/rolling/backup_7.tar.gz

Executing prebackup scripts
bash: ./scripts/backup_restore/pre_backup_complete.sh: No such file or directory
tar: ./services: Cannot stat: No such file or directory
tar: ./volumes: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
Rolling Overwritten: False
Backup Size (bytes): 45

Executing postbackup scripts
bash: ./scripts/backup_restore/post_backup_complete.sh: No such file or directory

Finished At: 2022-05-29T10-11-00

Items backed up:

./services/
./volumes/

Items Excluded:
 - No items

### End of log ###

The backup runs but it specifically says that there is 'no such file or directory' when it goes looking for the pre/post backup scripts.

I just can't get this to run and I feel like I'm missing something really obvious. Most of the IOTStack stuff is great as it just runs right out of the box.....but this one has me stumped. I suck at Linux so simple pointers would be appreciated. I have three IOTStack instances so getting the backups running would be good as I don't know how long I have before the SD cards will be fried. ;-)

Cheers!

[ukkopahis]

I've started to tackle the backup docs and implementation quirks and bugs. Haven't gotten around to testing my changes enough to be pull-requested, but this is a problem that will be fixed.

The problem being the backup.sh script is executed with the "wrong" current directory and bugs out.

Current Directory: /home/pi/IOTstack/scripts

This should be /home/pi/IOTstack. As a workaround, try running the script using the commands:

cd /home/pi/IOTstack
./scripts/backup.sh

[ChirpyTurnip]

Hi,

It is good that it is on the fix list. Running the script as suggested makes no difference - the script executes, claims to run the postback scripts but doesn't.

The backup.sh calls the pre/post scripts using:

bash ./backup_restore/post_backup_complete.sh >> $LOGFILE 2>&1

This is right, any attempt to change the path results in an error, so it is finding the pre/post scripts just fine. This means it's a problem with how the pre/post scripts are calling the post_backup.sh and pre_backup.sh files. So we take a look at the pre/post files:

#!/bin/bash

# This script runs any postbackup commands you may need

if [ -f "./post_backup.sh" ]; then
  echo "./post_backup.sh file found, executing:"
  bash ./post_backup.sh
fi

docker-compose up -d

I suspect the problem is in the relative path here as the script is actually being called by the parent process backup.sh in the directory above. To test this I amended the script to add "./backup_restore/" to the paths:

#!/bin/bash

# This script runs any postbackup commands you may need

if [ -f "./backup_restore/post_backup.sh" ]; then
  echo "./post_backup.sh file found, executing:"
  bash ./backup_restore/post_backup.sh
fi

docker-compose up -d

And now it works! So this is a simple fix for you I think....and one that even a knuckle dragger like me can make on my side to get it running until there is an update.

Although I suspect than when the update comes I will get all sorts of pesky errors again regarding uncommitted changes.

:-)

[ukkopahis]

It is good that it is on the fix list. Running the script as suggested makes no difference

You get the same "Cannot stat: No such file or directory" errors!?

The backup.sh calls the pre/post scripts using:

bash ./backup_restore/post_backup_complete.sh >> $LOGFILE 2>&1

Umm... what? Looking at https://github.com/SensorsIot/IOTstack/blob/master/scripts/backup.sh I can only find:

bash ./scripts/backup_restore/post_backup_complete.sh >> $LOGFILE 2>&1

Are you perhaps talking about your already modified backup.sh?

For the stock version of backup.sh, you're supposed to create pre_backup.sh or post_backup.sh directly into /home/pi/IOTstack. As evident by the backup script testing for their their existence for inclusion into the backup.

Even though a script file is located at scripts/backup_restore, any referenced files and paths in that script are relative to the current directory (=latest cd from before running the script).

[ChirpyTurnip]

My bad....it appears I didn't fully restore the default path after I messed around with it. But you point out something else:

For the stock version of backup.sh, you're supposed to create pre_backup.sh or post_backup.sh directly into /home/pi/IOTstack.

I had placed my post_backup.sh file in the backup_restore directory - so no doubt that will be why that wasn't getting picked up...I'll move my script tonight after work and re-run.

[ChirpyTurnip]

Moved the files to /IOTStack, reverted changes to backup.sh / post_backup_complete.sh. Now everything works. :-)

That means that the only real fix that is required is addressing the fact that enabling the backup/dropbox config doesn't actually result in files being uploaded.

Thanks for your help!

[Paraphraser]

Perhaps also take a look at IOTstackBackup and, in particular, the material at rclone (Dropbox). It walks through the Dropbox setup and you might get some other hints about how not to feel you're navigating blind when setting up Dropbox tokens.

[tomandegg]

Thanks @ChirpyTurnip, I ran into the same problem. Dropbox uploader working fine, nothing happening when running backup.sh. I assume the problem was introduced with this change: https://github.com/SensorsIot/IOTstack/commit/55539cc8a3323a466f0ce0f067d1a18d18ba4df9#diff-2481baed523de9f07631e555bc0df6ca4cc8c99ac550f9a7bb303813cb0ad378

The Dropbox part was completely removed with a - at least to me - unrelated commit "Removed influx script and old backup script". I'm not sure what the intention or future plan was.

Using the post_backup.sh hook solved the problem, however, I completely agree with you:

That means that the only real fix that is required is addressing the fact that enabling the backup/dropbox config doesn't actually result in files being uploaded.

I'd be willing to contribute, however, I'm only a few days into this specific project and I'm not sure about the state of the project. I'd appreciate if the maintainer could comment and communicate the status quo of the project and willingness to accept further development?

I see that there have been similar motivations, which makes me wonder whether this project ist still alive? https://github.com/SensorsIot/IOTstack/pull/570

[Paraphraser]

In my opinion (and it's nothing more than that), the core of IOTstack is excellent. By "core" I mean the service definitions and those parts of the menu system that help a first-time user assemble a docker-compose.yml.

Where IOTstack lacks a bit is managing service definitions. Things like temporarily disabling a service while keeping the state of play around so it can be enabled again.

Or getting any form of notification that the template upon which an active service definition was based, has changed on GitHub. Right now, any improvement made to a template will only get picked up by first-time users. Existing users will never find out unless they go looking.

Where IOTstack is pretty miserable (again, emphasis, this is just my opinion) is at the periphery. The scripts that handle installation (prerequisites like docker) are not fit for purpose, are scattered in multiple places for no reason I can discern, and are inconsistent.

The sentiment in the above paragraph is why I brought PiBuilder into being. Offline discussions as to approach didn't seem to be getting anywhere and I didn't want to wait.

And, indeed, far from the project being dead, this is still very much a live topic. Only a few days ago, Andreas (SensorsIot) wrote on the IOTstack Discord channel that "Getting Started" should be replaced with instructions to use PiBuilder. Slyke (who maintains this repo) has been talking about Ansible as an alternative.

The "backup" problem is something else that I put into the "pretty miserable" bucket. There were several reasons, including the need to take down non-copy-safe containers (like InfluxDB) for the duration of the backup. I wanted live backup. At the time there was no restore capability and I didn't see any sense in leaving it up to each user to roll their own. Plus, Dropbox being the only option and a bit flaky at that was irritating. That's why IOTstackBackup came into being.

I realise that none of this actually answers your question (or that of @ChirpyTurnip) directly. I can't answer because I don't know what was behind the changes. I do recall that Dropbox either made a transition to tokens or changed how tokens worked which caused a flurry of anxiety at the time. I remember thinking about that when I added Dropbox support to IOTstackBackup. I decided to return to first principles (ie follow the Dropbox doco, rather than copy what IOTstack had done). All I can really say is IOTstackBackup doesn't have this problem (which I can say because I use it multiple times every day via cron). Also, once the rclone "dropbox" remote is set up for IOTstackBackup, you can use rclone to do pretty much anything else involving moving stuff between your Pi and Dropbox.

So, if your objective is to do a deep dive into this area of IOTstack, figure out why Dropbox upload is not working, and fix it, then please do that. However, if your objective is just to get your backups off your Pi and safely stored on Dropbox then maybe IOTstackBackup will solve your immediate problem.

[tomandegg]

I agree with you. I was just referring to PRs like these: https://github.com/SensorsIot/IOTstack/pull/570/commits/7de8016faf16ab10a671feb40c6627b726c86f4a which try to at least remedy the situation, and as it hasn't been merged for over half a year, that made me wonder about the liveliness of the project, or the strategy that might be pursued in certain areas. I'm happy to offer my time if there is merit, but given my observations, I wanted to reach out first.

My objective wouldn't necessarily have been a deep dive into the project (but would have been fine if fruitful), I was just (and apparently not the first one) somewhat surprised to install the Dropbox integration via IOTstack to discover that it basically does nothing. I'll have a look at IOTstackBackup once I'll find some time to read the readme, thanks for the pointer. Backing up without having to take down the stack would absolutely be an advantage.

[Paraphraser]

Well, that link you provided is pointing to #570, right? That PR is still marked "untested - don't merge" by its author (Ukkopahis). Until Ukkopahis upgrades it to say it's tested and fit to be merged, I think the site's maintainer (Slyke) is probably well advised to leave it alone.

It's been a while since Ukkopahis last posted either here or on Discord. I just searched on Discord and the most recent post was October 26 last year. Unless I'm misinterpreting the GitHub activity page, it looks like there's nothing beyond the end of July last year. None of us has any way of knowing whether six months of silence is explained by a loss of interest in the project or something more untoward…

I can't speak for Ukkopahis but I gained the impression that submitting not-for-merge PRs was a way of announcing "be advised that I'm working in this area", to try to minimise merge conflicts. Personally, I try to avoid creating pull requests until I have tested everything and am happy for the PR to be reviewed and merged immediately (doesn't mean I don't make mistakes when I then have to rush to fix via another PR). Just different approaches to solving the same problem, I suppose.

If we make the assumption that Ukkopahis will never complete #570, I don't know the best way forward. That's likely because I don't know enough about Git and GitHub to understand the options. Brute force would be for Slyke to close the PR and someone else to start over from scratch. If there's a way for someone else to test and then contribute to the existing PR, I imagine Slyke could do whatever needs to be done on the administrative side to permit that.

Any ideas?

[tomandegg]

I don't know this small community well enough yet to make any educated guesses. However, as far as I can see PR#570 was labelled as "don't merge yet" by Slyke, and I don't see why. Maybe that was because of the inactivity of the author, maybe because of some design decisions, maybe something else.

I just observed that there are lots of open PRs, some of which solve problems which I encountered, too.

If @ukkopahis abandoned the project, one way forward would be to know why the PRs were labelled as "don't merge yet" so that we know whether somebody else should / could drive the PR to completion.

[Paraphraser]

I can't answer the general question of "lots of open PRs" but I agree it would be better if these were all finalised somehow.

Neither do I have any insight into whether Ukko has abandoned the project or if there's some other explanation.

As to #570, I agree that it was @Slyke who labelled it as "don't merge yet" but this is the marking I meant:

My guess is that Slyke added the label because of that marking.

But none of that gets us any closer to your goal.

In my previous post I said I didn't understand what options GitHub offered for managing this kind of problem. My Git/GitHub knowledge is right at the point of knowing just enough to be truly dangerous so I'm ultra-cautious.

I've just done some Googling and it led me to Assigning issues and pull requests to other GitHub users so it looks like it should be possible.

Perhaps review the PRs you're interested in and decide whether they can go in "as is" or need more work. For the former, add a comment of the "looks good to me" variety including an @Slyke with a request for the PR to be applied. For the latter, add a "please assign to me" comment (again @Slyke), and see what happens.

Or, if assignment doesn't work for some reason then start afresh with a new PR, explain that it's a substitute for the old PR and request the old PR be closed without being applied.