All local backup files are "Skipping already existing file" even if Dropbox folder is empty!

[IoTmessenger]

After a new install of this great add-on, it seems uploading the backup .tar files to Dropbox does not work for me. The add-on runs fine, it receives the command to upload, but treats all files as 'already existing' on Dropbox? Even if the Dropbox app folder is still completely empty.

I use add-on 1.2.0 on hass.io 0.71.0 (installed on a nuc with Ubuntu server 18.04, running in a docker).

I created a new app on Dropbox developers page for this and generated the token, following your instructions on Github.

My config for the add-on is:

{
  "oauth_access_token": "dedacted my access token...",
  "output": ""
}

The log file shows:

Log
starting version 3.2.4
[Info] Files will be uploaded to: /
[Info] Saving OAUTH_ACCESS_TOKEN to /etc/uploader.conf
[Info] Listening for messages via stdin service call...
{"command": "upload"}
[Info] Received message with command upload
[Info] Uploading all .tar files in /backup (skipping those already in Dropbox)
 > Skipping already existing file "/93d293ce.tar"
 > Skipping already existing file "/a98035c8.tar"
 > Skipping already existing file "/b1161847.tar"
 > Skipping already existing file "/b3254871.tar"
 > Skipping already existing file "/cdbf1233.tar"
 > Skipping already existing file "/f969d2cc.tar"
 > Skipping already existing file "/fa844957.tar"
 > Skipping already existing file "/fddc0a72.tar"

Any idea what is going on here?

[danielwelch]

Interesting, sorry about that. What happens if you set the output to a specific sub-folder?

[IoTmessenger]

I tried that first, but when that did not work I reset the config to defaults and just added my access token to test if it was something in my config. Did not help though. A reboot a hass.io did not help either.

[IoTmessenger]

I did some extra testing:

• Does not make a difference if I create a new app with a new access token in Dropbox, either with 'app folder access' or 'full Dropbox access'.
• Does not make a difference when I use a folder name or keep the folder name empty (default).
• Does not make a difference when that folder already exists or not, same outcome.
• I uninstalled the add-on, rebooted the server and re-installed the add-on, still same.

Maybe this helps:

I notice when I do not use an output folder, I see the log shows like:

Skipping already existing file "/93d293ce.tar"

But when I do use a folder output name, the log shows like this:

Skipping already existing file "/hassioBackup_fullDB/"

[edit] Also, when accessing the docker container of the add-on 'addon_7be23ff5_dropbox_sync', I see a copy of my /backup folder from hass.io share (with all my backup .tar files). I do not know if that is supposed to be copied there? Seems that that container for the add-on will get quite big.

[danielwelch]

@IoTmessenger I haven't been able to reproduce this, but from your description it sounds like the problem is with the dropbox_uploader.sh script side of things rather than the add-on. That's just my best guess.

The differences in the logs are expected.

As for your edit, you're probably just seeing the mounted /backup folder within the container, which is of course necessary for the add-on to work. If I'm misunderstanding your comment and you think there's another issue, please let me know.

As for troubleshooting your issue, it might be worth trying running the plain script outside of the context of the add-on to see what happens. The log output > Skipping already existing file "/93d293ce.tar (etc) is coming directly from the script, not from any code within the add-on. Whether you're able to run it on your device or otherwise, it's the only thing I can think of right now to try and pinpoint the problem.

[IoTmessenger]

Thanks for taking a look Daniel. I will test the script outside your add-on and when that still fails, I will contact Andrea. I will keep you updated.

[IoTmessenger]

I now have tested with the plain script from Andrea and found that it works as expected. I tested the following:

(Using SSH add-on made by frenck to access hass.io /config location)

---

Manual running the script 1st time:

---

I started with an empty Dropbox root folder using a newly created app with full permissions on the Dropbox.

?  /config ./dropbox_uploader.sh -s upload /backup/*.tar "/"

 This is the first time you run this script, please follow the instructions:

 1) Open the following URL in your Browser, and log in using your account: https://www.dropbox.com/developers/apps
 2) Click on "Create App", then select "Dropbox API app"
 3) Now go on with the configuration, choosing the app permissions and access restrictions to your DropBox folder
 4) Enter the "App Name" that you prefer (e.g. MyUploader3131940902517)

 Now, click on the "Create App" button.

 When your new App is successfully created, please click on the Generate button
 under the 'Generated access token' section, then copy and paste the new access token here:

 # Access token: <redacted>

 > The access token is <redacted>.
 Looks ok? [y/N]: y
   The configuration has been saved.

Then I ran the script manually using the following command:

?  /config ./dropbox_uploader.sh -s upload /backup/*.tar "/"
 > Uploading "/backup/7a379706.tar" to "/7a379706.tar"... DONE
 > Uploading "/backup/7f29763b.tar" to "/7f29763b.tar"... DONE
 > Uploading "/backup/93d293ce.tar" to "/93d293ce.tar"... DONE
 > Uploading "/backup/a98035c8.tar" to "/a98035c8.tar" by 10 chunks ........... DONE
 > Uploading "/backup/b1161847.tar" to "/b1161847.tar"... DONE
 > Uploading "/backup/b3254871.tar" to "/b3254871.tar"... DONE
 > Uploading "/backup/cdbf1233.tar" to "/cdbf1233.tar"... DONE
 > Uploading "/backup/e6b79caa.tar" to "/e6b79caa.tar"... DONE
 > Uploading "/backup/f969d2cc.tar" to "/f969d2cc.tar"... DONE
 > Uploading "/backup/fa844957.tar" to "/fa844957.tar"... DONE
 > Uploading "/backup/fddc0a72.tar" to "/fddc0a72.tar"... DONE
?  /config

So all files uploaded OK to the root of my Dropbox folder. Next scenario:

---

When using NO specific folder name in the config:

---

I created a new partial snapshot using node-red and your add-on and Log shows:

starting version 3.2.4
[Info] Files will be uploaded to: /
[Info] Saving OAUTH_ACCESS_TOKEN to /etc/uploader.conf
[Info] Listening for messages via stdin service call...
{"command": "upload"}
[Info] Received message with command upload
[Info] Uploading all .tar files in /backup (skipping those already in Dropbox)
 > Skipping already existing file "/7a379706.tar"
 > Skipping already existing file "/7f29763b.tar"
 > Skipping already existing file "/216ecb08.tar" --> newly created partial snapshot
 > Skipping already existing file "/93d293ce.tar"
 > Skipping already existing file "/a98035c8.tar"
 > Skipping already existing file "/b1161847.tar"
 > Skipping already existing file "/b3254871.tar"
 > Skipping already existing file "/cdbf1233.tar"
 > Skipping already existing file "/e6b79caa.tar"
 > Skipping already existing file "/f969d2cc.tar"
 > Skipping already existing file "/fa844957.tar"
 > Skipping already existing file "/fddc0a72.tar"

This was using the Config:

{
  "oauth_access_token": "<redacted>",
  "output": ""
}

So that did designate all files as already existing, even the new partial snapshot file "/216ecb08.tar".

When running the ./dropbox_uploader.sh script manual it skips all existing files (using -s parameter) and uploads the new file "/216ecb08.tar" OK:

?  /config ./dropbox_uploader.sh -s upload /backup/*.tar "/"
 > Skipping already existing file "/7a379706.tar"
 > Skipping already existing file "/7f29763b.tar"
 > Uploading "/backup/8d6c2326.tar" to "/216ecb08.tar"... DONE
 > Skipping already existing file "/93d293ce.tar"
 > Skipping already existing file "/a98035c8.tar"
 > Skipping already existing file "/b1161847.tar"
 > Skipping already existing file "/b3254871.tar"
 > Skipping already existing file "/cdbf1233.tar"
 > Skipping already existing file "/e6b79caa.tar"
 > Skipping already existing file "/f969d2cc.tar"
 > Skipping already existing file "/fa844957.tar"
 > Skipping already existing file "/fddc0a72.tar"
?  /config

---

When using a specific folder name in the config:

---

Using this config in the add-on:

{
  "oauth_access_token": "<redacted>",
  "output": "/backuptest/"
}

The output of the Log is:

starting version 3.2.4
[Info] Files will be uploaded to: /backuptest/
[Info] Saving OAUTH_ACCESS_TOKEN to /etc/uploader.conf
[Info] Listening for messages via stdin service call...
{"command": "upload"}
[Info] Received message with command upload
[Info] Uploading all .tar files in /backup (skipping those already in Dropbox)
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"
 > Skipping already existing file "/backuptest/"

I created the new empty folder manually in Dropbox first, but no files were uploaded using the add-on. Then I tried manually initiating the .dropboxuploader.sh script to the same new dropbox folder:

?  /config ./dropbox_uploader.sh -s upload /backup/*.tar "/backuptest/"
 > Uploading "/backup/216ecb08.tar" to "/backuptest/216ecb08.tar"... DONE
 > Uploading "/backup/7a379706.tar" to "/backuptest/7a379706.tar"... DONE
 > Uploading "/backup/7f29763b.tar" to "/backuptest/7f29763b.tar"... DONE
 > Uploading "/backup/93d293ce.tar" to "/backuptest/93d293ce.tar"... DONE
 > Uploading "/backup/a98035c8.tar" to "/backuptest/a98035c8.tar" by 10 chunks ........... DONE
 > Uploading "/backup/b1161847.tar" to "/backuptest/b1161847.tar"... DONE
 > Uploading "/backup/b3254871.tar" to "/backuptest/b3254871.tar"... DONE
 > Uploading "/backup/cdbf1233.tar" to "/backuptest/cdbf1233.tar"... DONE
 > Uploading "/backup/e6b79caa.tar" to "/backuptest/e6b79caa.tar"... DONE
 > Uploading "/backup/f969d2cc.tar" to "/backuptest/f969d2cc.tar"... DONE
 > Uploading "/backup/fa844957.tar" to "/backuptest/fa844957.tar"... DONE
 > Uploading "/backup/fddc0a72.tar" to "/backuptest/fddc0a72.tar"... DONE
?  /config

So there seems to be something not working as it should using the add-on. The output in the Log is also different as the output from running the script manually. It could be that it is caused by me running Hass.io on a nuc with Ubuntu server 18.04, running in a docker. The paths to the locations of files are maybe different?

[danielwelch]

It could be that it is caused by me running Hass.io on a nuc with Ubuntu server 18.04, running in a docker.

That’s a good guess for what’s going on, but I’m not sure. I didn’t realize this earlier, but when you’re using the output folder setting, something is going wrong. You’ll notice that in the logs, the Dropbox uploader script is complaining that the folder already exists, not the specific .tar file. I’m not sure why, and it still doesn’t explain what’s going wrong in the “top level” example you gave. I’d investigate your hassio file structure and make sure it’s standard. I apologize I don’t have time to troubleshoot further right now. On one hand I wonder if there’s some sort of docker within docker problem that you’re running into or other issue with running as a docker container on Ubuntu , but on the other hand, just running the script within the SSH add-on (also a docker container within docker container) seems to work alright.

[IoTmessenger]

Maybe I have put it wrong, it is not installed as a docker in a docker, I just followed the instructions on the official homeassistant hass.io installation guide: https://www.home-assistant.io/hassio/installation/ (Alternative: install on generic Linux server).

So all paths on my installation should be as default for hass.io I presume?

/edit After installing, I used a snapshot of my old RPI's Hass.io installation to restore my config.

[danielwelch]

I understand, my fault. A docker container (add-on) inside virtual machine then it looks like. Have you tried running the add-on on a fresh Hass.io, with your credentials, not restored from a rpi configuration?

[IoTmessenger]

I will try to make some time to test the add-on on a fresh install.

[matisaul]

Hi @IoTmessenger and @danielwelch , I'm having this issue too, with the same Hassio config (Hassio over docker in Ubuntu).

[danielwelch]

Thanks @matisaul, I think we’ve found the cause. I’ll try and find a solution when I can.

[IoTmessenger]

Little update, found some time to make a fresh install.

I installed a fresh Ubuntu server with hass.io without using my old config snapshot, still having the same issue.

Also tested the add-on using the latest HassOS in a VM (VirtuaBox) on a fresh installation using the VMDK disk image. For instructions how to use the vmdk, see: link.

That works OK with the exact same config for the add-on.

[iceman73]

+1 for having the same issue on Ubuntu VM running hassio

[01kbpatel]

Also having same issue addon works with pi installation but fails with nuc installation

[nigel914]

+1 having the same issue on an intel nuc also.

[d0ugal]

I hit the same problem as y'all, so I threw together a simpler Python based hass.io addon. I don't mean to plug here, but if you are blocked from using this one then it might help you out. https://github.com/d0ugal/hassio-dropbox-upload

I don't have as many features, but I have some plans (see the issues). Contributions are also welcome. I have only been using it for a few days but it has worked for me. I also wanted something Python based so I could more easily work on it over time.

[IoTmessenger]

I can confirm that the addon d0ugal has created, works for me as well. I am able to upload to Dropbox OK in an fully automated manner. This is a valid solution for users with an Intel NUC running docker and Hass.io.

[danielwelch]

@d0ugal thanks for sharing! No worries about plugging, only goal I have is for the Hass.io ecosystem to be better. I’ve been thinking of transitioning away from the bash script and to a Python based approach as well for the sake of extra features.

[dougal]

FAO @d0ugal. :)

[adrianmihalko]

I have same problem.