No longer backing up after 1.6 on Unraid

[rjbutler]

I've been using this project since last summer and it's been working great.

I updated several docker images on my Unraid server a couple of weeks ago and I guess I got complacent because I didn't verify that Backblaze was running. I noticed this morning that it hadn't backed up since Jan 29, and when I tried to connect to the web interface I got the blank screen others have mentioned. Following the advice given, I switched to the ubuntu22 tag and it appeared to be running correctly.

However, it isn't backing anything up. If I click "Backup Now", this status changes to "Producing File Lists", but then it finishes and nothing new gets backed up despite the fact that there are files that have changed. I also tried adding a new drive that had not previously been backed up and that too failed to back up.

The issues tab says

SkippedFilesReportStarted: 2024-01-29 19:24:13

No Issues Found.

Which would seem to indicate that the backup didn't run.

Any ideas? Thank you

ls -al /config/wine/dosdevices/ lrwxrwxrwx 1 app app 10 Jul 6 2023 c: -> ../drive_c lrwxrwxrwx 1 app app 9 Jul 6 2023 e: -> /drive_e/ lrwxrwxrwx 1 app app 9 Aug 6 2023 h: -> /drive_h/ lrwxrwxrwx 1 app app 9 Jul 13 2023 i: -> /drive_i/ lrwxrwxrwx 1 app app 9 Jul 10 2023 m: -> /drive_m/ lrwxrwxrwx 1 app app 9 Jul 8 2023 p: -> /drive_p/

Is there a way to force a rescan? I think on Windows you hold down Alt and click the restore button, but that doesn't seem to work with Wine.

[rjbutler]

A rescan seems to have fixed it. At least on a Mac, Ctrl plus clicking the restore button caused a rescan.

[mason1171]

A rescan seems to have fixed it. At least on a Mac, Ctrl plus clicking the restore button caused a rescan.

Im experiencing the same issue but it looks like my rescan is stuck. I think this is related to the large file upload system @JonathanTreffler @traktuner What info can I pass to yall? I recently had to play around with my config folder/docker perms to fix that black screen issue and since then backblaze hasn't uploaded.

[traktuner]

A rescan seems to have fixed it. At least on a Mac, Ctrl plus clicking the restore button caused a rescan.

Im experiencing the same issue but it looks like my rescan is stuck. I think this is related to the large file upload system @JonathanTreffler @traktuner What info can I pass to yall? I recently had to play around with my config folder/docker perms to fix that black screen issue and since then backblaze hasn't uploaded.

A rescan could take a very long time, depending on your backup set (If you are backing up millions of little files, that may take days).

Could you please paste the full docker container logs here and let us know which Backlaze version is installed? (docker logs backblaze-personal-wine-container) Also, please paste the log output from this file here: [your-config-folder]/dosdevices/c:/backblaze-wine-startapp.log

And your docker run command including all environment variables. Thank you!

[mason1171]

9.0.0.749 _Backblaze_Personal_Backup_logs (1).txt backblaze-wine-startapp.log

docker run -d --name='Backblaze_Personal_Backup' --net='bridge' -e TZ="America/New_York" -e HOST_OS="Unraid" -e HOST_HOSTNAME="NAS" -e HOST_CONTAINERNAME="Backblaze_Personal_Backup" -e 'SECURE_CONNECTION'='0' -e 'TZ'='EST' -e 'DISPLAY_WIDTH'='660' -e 'DISPLAY_HEIGHT'='476' -e 'VNC_PASSWORD'='' -e 'GROUP_ID'='0' -e 'USER_ID'='0' -e 'X11VNC_EXTRA_OPTS'='' -e 'ENABLE_CJK_FONT'='' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.webui='http://[IP]:[PORT:5800]/' -l net.unraid.docker.icon='https://www.backblaze.com/pics/cloud-blaze.png' -p '5800:5800/tcp' -p '5900:5900/tcp' -v '/mnt/user/':'/drive_d/':'rw' -v '/mnt/user/appdata/backblaze_personal_backup/':'/config/':'rw' --init 'tessypowder/backblaze-personal-wine:latest' Thank you

[traktuner]

Thanks @mason1171 ! Could you please try it with the :ubuntu22 (or 20) tag instead of latest? We pushed out some fixes over the last week and maybe this also fixes your issue. Your Backblaze version will also be updated. The installer will pop up when you start the container, please click install. Please let me know if that helps!

[mason1171]

Thanks @mason1171 ! Could you please try it with the :ubuntu22 (or 20) tag instead of latest? We pushed out some fixes over the last week and maybe this also fixes your issue. Your Backblaze version will also be updated. The installer will pop up when you start the container, please click install. Please let me know if that helps!

Just updated. New Backblaze showing Producing file lists so Ill keep you updated. Also, I see this version has a version history level indicator so I hope that means that the version retention history'll be respected on the backend now. Previously, the behavior in my instance was my files were being deleted despite having the year retention policy.

No luck still zero remaining files despite that not being true @traktuner

[JonathanTreffler]

Please be patient with rescans, they take ages and give ZERO clues, that they are running in the background in the user interface. If you need to know for sure you can find them in the bztransfer logs (in the config directory, not the docker logs)

[mason1171]

Please be patient with rescans, they take ages and give ZERO clues, that they are running in the background in the user interface. If you need to know for sure you can find them in the bztransfer logs (in the config directory, not the docker logs)

I see. I was expecting increased disk activity. Ive started a new rescan.

[mason1171]

Please be patient with rescans, they take ages and give ZERO clues, that they are running in the background in the user interface. If you need to know for sure you can find them in the bztransfer logs (in the config directory, not the docker logs)

Where exactly are those bztransfer logs? I fuzzy searched the config dir for bztransfer and nothing was found @JonathanTreffler

[JonathanTreffler]

Please be patient with rescans, they take ages and give ZERO clues, that they are running in the background in the user interface. If you need to know for sure you can find them in the bztransfer logs (in the config directory, not the docker logs)

Where exactly are those bztransfer logs? I fuzzy searched the config dir for bztransfer and nothing was found @JonathanTreffler

I got the name wrong, you are looking for the log files in the bzfilelist and bztramsmit folders in /config/wine/dosdevices/c:/ProgramData/Backblaze/bzdata/bzlogs (or similar, I reconstructed it for this message, but don't have an instance to test this path)

[mason1171]

Thanks, ill take another look

On Feb 12, 2024, 3:47 AM -0500, Jonathan Treffler @.***>, wrote:

Please be patient with rescans, they take ages and give ZERO clues, that they are running in the background in the user interface. If you need to know for sure you can find them in the bztransfer logs (in the config directory, not the docker logs) Where exactly are those bztransfer logs? I fuzzy searched the config dir for bztransfer and nothing was found @JonathanTreffler I got the name wrong, you are looking for the log files in the bzfilelist and bztramsmit folders in /config/wine/dosdevices/c:/ProgramData/Backblaze/bzdata/bzlogs (or similar, I reconstructed it for this message, but don't have an instance to test this path) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[mason1171]

Please be patient with rescans, they take ages and give ZERO clues, that they are running in the background in the user interface. If you need to know for sure you can find them in the bztransfer logs (in the config directory, not the docker logs)

Where exactly are those bztransfer logs? I fuzzy searched the config dir for bztransfer and nothing was found @JonathanTreffler

I got the name wrong, you are looking for the log files in the bzfilelist and bztramsmit folders in /config/wine/dosdevices/c:/ProgramData/Backblaze/bzdata/bzlogs (or similar, I reconstructed it for this message, but don't have an instance to test this path)

The rescan worked! @JonathanTreffler @traktuner Thank you

[cookandy]

I'm running into something similar using Unraid and the latest ubuntu22 image, pulled today.

If I select continuous backup, it will sit on a screen that says "Initial Backup Paused", even though this is a full backup that has completed in the past. If I select "Backup Now" it will say "Initial Backup in Progress" and will scan and upload the diffs, and then goes back to "Initial Backup Paused".

If I select Only backup when I select <backup> it will rescan the entire filesystem (which takes hours), upload the diffs, and then go back to "Initial Backup Paused".

I can't seem to break out of this "initial backup" sync loop, and make it show a completed backup. In the logs I see

2024-02-13 17:36:43 - STARTBACKUP
2024-02-13 17:38:45 - END_BACKUP: Backed up 0 FILES / 0 KB

It's worth mentioning that I inherited this backup from a previous install after switching to ubuntu22. Not sure if that makes a difference.

Any ideas? Thanks for the great project!

[traktuner]

@cookandy Normally, you'll never need to inherit your backup. You can switch docker tags around if you like, but that has nothing to do with Backblaze itself. Backblaze and it's data are on your local (unraid) disk, not in the container. Of course that's needed when you completely throw away your Backblaze installation. Are you backing up a folder which is a "cloud only" folder? Like files are dummys on disk and need to get pulled from cloud to be available? That can for instance cause the "Initial Backup Paused" notification. Can you check if there are files listed in the "Issues" tab in Backblaze settings?

[cookandy]

Normally, you'll never need to inherit your backup.

Yeah not sure what happened, but at some point permissions got messed up with the Docker image and it would lock the btrfs filesystem (the one Docker and all the app use), and make the filesystem read-only as soon as I started the container. I think something changed with the way permissions were handled. The only way I was able to fix it was to completely wipe the Backblaze install (including the .bzvol) and start fresh. The inherit seemed to work at first, and it showed "Backup complete". But at some point it fell into this sync loop.

I thought the same about the files with "issues". I looked through the list and most of them were locked files such as DB pointers/locks, etc. I went ahead and added exclusions for every folder with locked files, so now the issues tab "no issues" – but I'm still in this initial sync loop.

To be clear, it does appear that during this initial sync loop that new things are in fact getting backed up. It will just never say "backup complete". 🤷

[traktuner]

Interesting. Would be good to have a look at the bzlogs folder. Will figure out later which log is important there. The backup complete message is a bit "strange" because I guess it's hardcoded and will get displayed when the files to be backed up are less than 1gb (or something like that). Maybe we'll figure that out on the logs.

[cookandy]

There's a decent amount of sensitive info in the logs (files, paths, etc.). Do you know which log we should start with? I can try to anonymize it a bit and send it over.

[inittabayanami]

been trying to follow recommendations here, switched to ubuntu22 tag, it tries to install a new version, but the gui for the installer never shows up (see screenshot) if i click on the installer in the task bar, I get nothing, if I try to right click and 'move' or 'restore' the app will just close, then the desktop will disappear. Any thoughts? this is on truenas scale kubernetes, I've attached the container yaml as well which should mimic a docker run command, can also open a seperate issue if this is off track.

Thanks! backblaze deploy.txt

[traktuner]

it tries to install a new version, but the gui for the installer never shows up

Hey, Can you try to restart the container? What happens when you click on the installer tab a few times? Thank you

[traktuner]

There's a decent amount of sensitive info in the logs

I know, I des not mean send over all the logs 😅 I will figure out tomorrow which log is important here.

[inittabayanami]

it tries to install a new version, but the gui for the installer never shows up

Hey, Can you try to restart the container? What happens when you click on the installer tab a few times? Thank you

Multiple clicks does nothing, the app in the task bar just remains blue. restarting leads me back to the same start bar with the installer running but no installer showing

[traktuner]

it tries to install a new version, but the gui for the installer never shows up

Hey, Can you try to restart the container? What happens when you click on the installer tab a few times? Thank you

Multiple clicks does nothing, the app in the task bar just remains blue.

restarting leads me back to the same start bar with the installer running but no installer showing

Thanks for letting me know! I will test this tomorrow.

[cookandy]

I will figure out tomorrow which log is important here.

Thank you! I grepped through the bzlogs directory for "paused", but nothing came up. However, when I look in the latest bztransmit.log, I do see about 70 warnings about files not existing, even though they do

WARNING: skipping_AA file because it no longer exists: <file>

And then towards the end of the log I see it complain about there still being too many files to declare victory

2024-02-14 22:55:43 0000001872 - ATTEMPTING TO DECLARE VICTORY on initial upload processId=1872 - after ProduceTodoListOfFilesToBeBackedUp, numFilesRemaining=77, numBytesRemaining=4223017993
2024-02-14 22:55:44 0000001872 - BackupSummaryUserWouldSee=Selected_xxx__files_/_xxx_MB__Remaining_77_files_/_4,027_MB
2024-02-14 22:55:44 0000001872 - NOT DECLARING VICTORY on initial upload - too much left

I'm not sure why these files are showing as no longer existing. I've done a full rescan several times, and the files it is referencing haven't changed. I notice most of the files have a question mark (?) or quote (") in the filename, but not all of them. Perhaps it's complaining about the those characters.

I can try to add some more exclusions and/or rename those files to see if i can get it to complete.

[traktuner]

I've done a full rescan several times

Hey @cookandy , please be always careful with rescans and do everything else first to solve any issues (checking logs, etc). Rescans can take a very long time, and will happen automatically in the background, because that's how Backblaze discovers new files to be backed up. A rescan is only necessare in very specific and rare situations.

So the logs tell us what I mentioned above, it's about file size and count, that it can't "declare victory" for the initial backup run. Could you please ssh into your container and check the specific mounts if you can access those files from within the container? docker exec --user app backblaze_personal_backup wine explorer

If the files are not there you have a permission issue with the file on your disk.

[traktuner]

it tries to install a new version, but the gui for the installer never shows up

Hey , Can you try to restart the container? What happens when you click on the installer tab a few times? Thank you

Multiple clicks does nothing, the app in the task bar just remains blue. restarting leads me back to the same start bar with the installer running but no installer showing

Hey @inittabayanami, can you try to right-click the installer in the task bar and then click "Restore"? Thank you

[inittabayanami]

it tries to install a new version, but the gui for the installer never shows up

Hey , Can you try to restart the container? What happens when you click on the installer tab a few times? Thank you

Multiple clicks does nothing, the app in the task bar just remains blue. restarting leads me back to the same start bar with the installer running but no installer showing

Hey @inittabayanami, can you try to right-click the installer in the task bar and then click "Restore"? Thank you

I had tried that before and it had just closed, I went to try it this morning after the container had been running since yesterday and restore wasn't an option but 'move' was, I clicked it and the installer then popped up.

I'm wondering if this is just a time thing, multiple minutes (10-20 at times) had passed yesterday when trying things and nothing would bring the installer up, but clicking move this morning worked.

[cookandy]

@traktuner thanks for the tips. I've confirmed the files are indeed present in the wine explorer, but as I mentioned most of them have either quotes or question marks in the names. I'll spend some time today renaming some of those files to see if it makes a difference.

[cookandy]

Just wanted to close the loop on this.

Seems like my issue was related to btrfs filesystem corruption. I was cleaning up a previous Backblaze install and anytime I tried to remove the backblaze-personal-backup/wine/drive_c/ProgramData/Backblaze/bzdata directory the filesystem would lock and go into read only mode, with these errors from the kernel

Feb 16 15:55:50 kernel: BTRFS: error (device dm-7: state A) in __btrfs_free_extent:3072: errno=-2 No such entry
Feb 16 15:55:50 kernel: BTRFS info (device dm-7: state EA): forced readonly
Feb 16 15:55:50 kernel: BTRFS error (device dm-7: state EA): failed to run delayed ref for logical 154499358720 num_bytes 8192 type 178 action 2 ref_mod 1: -2
Feb 16 15:55:50 kernel: BTRFS: error (device dm-7: state EA) in btrfs_run_delayed_refs:2149: errno=-2 No such entry

The odd thing is, this happened with both of my BackBlaze install directories, but no other files on the filesystem. 🤷

I was able to repair the filesystem, which also repaired all the WARNING: skipping_AA file because it no longer exists: <file> errors. I was then able to re-install and inherit my backup without issues. Everything appears to be backing up fine now.

Thanks for your help troubleshooting!