search

JonathanTreffler / backblaze-personal-wine-container

Run the Backblaze personal backup client in a docker container
https://hub.docker.com/r/tessypowder/backblaze-personal-wine
GNU Affero General Public License v3.0
374 stars 35 forks source link

Stuck at 15mbit Upload #127

Closed newoski closed 6 months ago

newoski commented 6 months ago

I'm stumped and never had this problem until a week or so ago. No matter what changes I make to the Performance tab, I am stuck at 15mbit per second upload -- regardless of the file size (ie upload speed of a 4GB video is still only 15mbit total). My connection is gigabit and if I run a speedtest from my Unraid server, I get between 750mbit and 900mbit, so it's not a network issue, so far as I can tell.

Backblaze used to completely push my system and use like 99% of CPU, pretty constantly, while uploading. Not I'm seeing much lower usage. No idea what changes or how to fix it

traktuner commented 6 months ago

Hey @newoski I'd say that this is hard to diagnose. It depends on so many factors: threads you selected for backup and your Backblaze client settings, current Bandwith constraints at Backblaze datacenter itself, etc.

Do you have a symmetrical gigabit connection (1gbit down / 1gbit up)? You can then set the threads to 100, if the file is large enough (at least 10GB) it should saturate your upload. This also depends on your CPU and RAM. The more threads, the more CPU/RAM will be used. If the initial backup is done, Backblaze scans your drives for new files and puts it in the upload queue (small files first). So If you have many small files to back up, the upload will be very slow.

So - "what changes and how to fix" is hard to answer. Is your initial backup already done?

newoski commented 6 months ago
  1. Initial backup is not done
  2. Current CPU usage fluctuates second to second between like 30-60%. It used to sti pretty much above 90% when it was uploading faster in the past
  3. RAM usage is at about 20% of 32GB
  4. None of the slow files are small -- if they were, I would assume that was the cause. They're all over a gig, some as big as 70GB
  5. Symmetrical connection of about 350mbit up/down. I will occasionally see a gigabit up/down when speed testing via unraid, but I'm only paying for 350/350 and I get 350/350 pretty consistently
  6. Gigabit ethernet connection and if I do a LAN speed test I'm seeing gigabit speeds
  7. I've tried Inherit Backup and fresh backups. Both experience the slow speeds with large files

On Tue, Feb 20, 2024 at 9:55 AM Thomas @.***> wrote:

Hey @newoski https://github.com/newoski I'd say that this is hard to diagnose. It depends on so many factors: threads you selected for backup and your Backblaze client settings, current Bandwith constraints at Backblaze datacenter itself, etc.

Do you have a symmetrical gigabit connection (1gbit down / 1gbit up)? You can then set the threads to 100, if the file is large enough (at least 10GB) it should saturate your upload. This also depends on your CPU and RAM. The more threads, the more CPU/RAM will be used. If the initial backup is done, Backblaze scans your drives for new files and puts it in the upload queue (small files first). So If you have many small files to back up, the upload will be very slow.

So - "what changes and how to fix" is hard to answer. Is your initial backup already done?

— Reply to this email directly, view it on GitHub https://github.com/JonathanTreffler/backblaze-personal-wine-container/issues/127#issuecomment-1954392341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJGHKXNWBJVAKHKZPID333YUS2NRAVCNFSM6AAAAABDPYME5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJUGM4TEMZUGE . You are receiving this because you were mentioned.Message ID: <JonathanTreffler/backblaze-personal-wine-container/issues/127/1954392341@ github.com>

newoski commented 6 months ago

Sorry, forgot to mention... changing threads has zero impact on this, and the 15mbit is constant around the clock all day long, so I don't think it's a data center issue

On Tue, Feb 20, 2024 at 10:38 AM Owen Sakovics @.***> wrote:

  1. Initial backup is not done
  2. Current CPU usage fluctuates second to second between like 30-60%. It used to sti pretty much above 90% when it was uploading faster in the past
  3. RAM usage is at about 20% of 32GB
  4. None of the slow files are small -- if they were, I would assume that was the cause. They're all over a gig, some as big as 70GB
  5. Symmetrical connection of about 350mbit up/down. I will occasionally see a gigabit up/down when speed testing via unraid, but I'm only paying for 350/350 and I get 350/350 pretty consistently
  6. Gigabit ethernet connection and if I do a LAN speed test I'm seeing gigabit speeds
  7. I've tried Inherit Backup and fresh backups. Both experience the slow speeds with large files

On Tue, Feb 20, 2024 at 9:55 AM Thomas @.***> wrote:

Hey @newoski https://github.com/newoski I'd say that this is hard to diagnose. It depends on so many factors: threads you selected for backup and your Backblaze client settings, current Bandwith constraints at Backblaze datacenter itself, etc.

Do you have a symmetrical gigabit connection (1gbit down / 1gbit up)? You can then set the threads to 100, if the file is large enough (at least 10GB) it should saturate your upload. This also depends on your CPU and RAM. The more threads, the more CPU/RAM will be used. If the initial backup is done, Backblaze scans your drives for new files and puts it in the upload queue (small files first). So If you have many small files to back up, the upload will be very slow.

So - "what changes and how to fix" is hard to answer. Is your initial backup already done?

— Reply to this email directly, view it on GitHub https://github.com/JonathanTreffler/backblaze-personal-wine-container/issues/127#issuecomment-1954392341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJGHKXNWBJVAKHKZPID333YUS2NRAVCNFSM6AAAAABDPYME5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJUGM4TEMZUGE . You are receiving this because you were mentioned.Message ID: <JonathanTreffler/backblaze-personal-wine-container/issues/127/1954392341 @github.com>

traktuner commented 6 months ago

Thanks for the follow-up. We don't limit the bandwith in any way. Changing threads should definetely have an impact on the upload speed. Also keep in mind, that peering partners of your internet provider can change over time. That means, if your internet provider "cuts a cord" to a peering partner, like a big internet exchange, your files may have to be re-routed to many more exchanges to reach Backblaze, which will also slow down the transfer speed. You can also try to install Backblaze without the docker container on a PC or macOS device, or a virtual Windows machine just to do some more bandwith-testing. To test the internet throughput of your docker host, you can also use a project like speedtest-tracker which measures agains ookla's speedtest.net - if this gives you the expected 350/350 speeds, then I would assume that this is a - maybe temporarly - Backblaze issue

Please do some more testing and let us know what you find. Unfortunately, I have no way in replicating this behaviour, also no related issues were reported.

newoski commented 6 months ago

docker run --rm --net=host tianon/speedtest --accept-license resulted in about 350 up and down, matching my FIOS plan I'm stumped. I guess I'll just limp along at this speed, assuming it's a backblaze issue. I'll reach out to their support, as well. thanks for the assistance

d3fr0st commented 6 months ago

Also seeing extremely similar behaviour on my container on unraid with :latest. Seeing normal speed on my desktop backup, on the same network.

itzmichael123 commented 6 months ago

I have noticed the same behaviours recently (also unraid) since the last week/week and half too. Went from around 600GB/day to 50GB/day max if that.. Network on my side is fine too. Also seeing a lot lower system usage vs before this where it would better use a lot more resources

traktuner commented 6 months ago

Hey, thank you for your reports! Would be great if you can share the Backblaze version you're using! There is a new client available which I'm testing right now (9.0.1.767) and maybe this fixes that issue. I still don't think that this is a problem with this docker container image. I'll try to push out a release with the new client next week, if it's stable for me. (for me, 9.0.1.767 maxes out my 100Mbit/s upload speed) You can update to the latest version when you set FORCE_LATEST_UPDATE="true" via environment variable / docker run command. Please also note that this is at your own risk and totally unsupported by us when you install a version which is not declared as known-good by us. If a version is declared stable, we'll update the docker image and it automatically downloads and installs the upate.

newoski commented 6 months ago

Happy to share that, but where do I access the info? Clicking the ? in the top right of the app UI doesn't do anything. Is there a command I can send the docker to generate version number?

traktuner commented 6 months ago

If you are using the latest version of our docker container you should see a windows-like environment with a task bar. when you click in the bottom right corner of the task bar, the Backblaze menu opens - click "About..." - there you can find the version

newoski commented 6 months ago

9.0.1.763 (20240119234549)

traktuner commented 6 months ago

9.0.1.763 (20240119234549)

Great, then this is our current known-good release. Hopefully next week I'll have the time to push out a new release of this container and then your Backblaze client should update automatically to 9.0.1.767

daveerickson commented 6 months ago

Hey, thank you for your reports! Would be great if you can share the Backblaze version you're using! There is a new client available which I'm testing right now (9.0.1.767) and maybe this fixes that issue. I still don't think that this is a problem with this docker container image. I'll try to push out a release with the new client next week, if it's stable for me. (for me, 9.0.1.767 maxes out my 100Mbit/s upload speed) You can update to the latest version when you set FORCE_LATEST_UPDATE="true" via environment variable / docker run command. Please also note that this is at your own risk and totally unsupported by us when you install a version which is not declared as known-good by us. If a version is declared stable, we'll update the docker image and it automatically downloads and installs the upate.

If I force latest update and have issues, is it possible to roll back?

traktuner commented 6 months ago

Hey, thank you for your reports! Would be great if you can share the Backblaze version you're using! There is a new client available which I'm testing right now (9.0.1.767) and maybe this fixes that issue. I still don't think that this is a problem with this docker container image. I'll try to push out a release with the new client next week, if it's stable for me. (for me, 9.0.1.767 maxes out my 100Mbit/s upload speed) You can update to the latest version when you set FORCE_LATEST_UPDATE="true" via environment variable / docker run command. Please also note that this is at your own risk and totally unsupported by us when you install a version which is not declared as known-good by us. If a version is declared stable, we'll update the docker image and it automatically downloads and installs the upate.

If I force latest update and have issues, is it possible to roll back?

Yes, but that is a manual task (download the pinned bz version you find in our repository, and then executing the install commands manually which you can find in the startapp.sh script).

newoski commented 6 months ago

I'm on 767 now. Not seeing any improvement

traktuner commented 6 months ago

I'm on 767 now. Not seeing any improvement

Are you on the US or EU region for your Backblaze backup location?

newoski commented 6 months ago

How do I check that?

traktuner commented 6 months ago

How do I check that?

You need to specify this when you register. There is a drop town menu "Region" on the register page where you can select EU or US. I guess US is default if you did not make a change. My endpoint is EU, and I don't see any issues with my Backup speed on Version .767 So probably really a Backblaze issue. Sorry to say but I think there is nothing we can do from our end.

echoblag commented 6 months ago

Howdy, I'm relieved to see this thread has started. I spent a day diagnosing the same speed related issue last week. I had received an email that no backups had run for two weeks, prompting me to investigate.

To summarize it looks like I'm getting speeds closer to 7 Mbps or 700 KB/s 12 Mbps or 1.5 MB/s.

First it seems my container running version 8 on the main tag stopped backing up, this container is regularly updated.

Internet Connection Details

My Internet connection is symmetrical 1 Gbps fibre. Below are the daily speed test results against two different servers. You can even see the upload dip as it was competing with the backups catching up around 16th February.

image

I am backing up to the EU data centre.

Tags and Versions

Pre and post speed issue comparison

Comparing bandwidth usage and upload speeds to before and after the change.

Made some edits to the speed calculation. The backlog of 250GB took somewhere over a day to catch up, which implies a speed of somewhere between 10-15 Mbps.

cadvisor based metrics screenshot included below image

Transfer speeds before this issue started.

image

Focusing on just the back up time we can see the prior 15 MB/s speeds

image
newoski commented 6 months ago

Is it possible to rollback to whatever version was working? Do we know which that would be?

itzmichael123 commented 6 months ago

Was on 9.0.1.763 now on 9.0.1.767 and seems to be having the same issue but ill give it some time to see how it is. I'm on the EU region

Edit: Still just as bad, stuck at 5-6Mbit/s

newoski commented 6 months ago

It sounds like v8 was working

On Thu, Feb 22, 2024 at 11:26 AM itzmichael123 @.***> wrote:

Was on 9.0.1.763 now on 9.0.1.767 and having same issue with EU region

— Reply to this email directly, view it on GitHub https://github.com/JonathanTreffler/backblaze-personal-wine-container/issues/127#issuecomment-1959809255, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJGHKREM5DGKTR2VQSQSFLYU5WT3AVCNFSM6AAAAABDPYME5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZHAYDSMRVGU . You are receiving this because you were mentioned.Message ID: <JonathanTreffler/backblaze-personal-wine-container/issues/127/1959809255@ github.com>

echoblag commented 6 months ago

It sounds like v8 was working … On Thu, Feb 22, 2024 at 11:26 AM itzmichael123 @.***> wrote: Was on 9.0.1.763 now on 9.0.1.767 and having same issue with EU region — Reply to this email directly, view it on GitHub <#127 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJGHKREM5DGKTR2VQSQSFLYU5WT3AVCNFSM6AAAAABDPYME5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZHAYDSMRVGU . You are receiving this because you were mentioned.Message ID: <JonathanTreffler/backblaze-personal-wine-container/issues/127/1959809255@ github.com>

I don't have logs that go back that far, but version 8 might be the one that stopped backing up for me.

newoski commented 6 months ago

Ya, I mean, right now I would say the Backup functionality is dead. There are really two issues here:

On Thu, Feb 22, 2024 at 12:12 PM B Ivanov @.***> wrote:

It sounds like v8 was working … <#m1426103758884293420> On Thu, Feb 22, 2024 at 11:26 AM itzmichael123 @.***> wrote: Was on 9.0.1.763 now on 9.0.1.767 and having same issue with EU region — Reply to this email directly, view it on GitHub <#127 (comment) https://github.com/JonathanTreffler/backblaze-personal-wine-container/issues/127#issuecomment-1959809255>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJGHKREM5DGKTR2VQSQSFLYU5WT3AVCNFSM6AAAAABDPYME5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZHAYDSMRVGU . You are receiving this because you were mentioned.Message ID: < /issues/127 https://github.com/JonathanTreffler/backblaze-personal-wine-container/issues/127 /1959809255@ github.com>

I don't have logs that go back that far, but version 8 might be the one that stopped backing up for me.

— Reply to this email directly, view it on GitHub https://github.com/JonathanTreffler/backblaze-personal-wine-container/issues/127#issuecomment-1959897962, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJGHKT7GNXPYWHAR3SYPYLYU54AVAVCNFSM6AAAAABDPYME5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZHA4TOOJWGI . You are receiving this because you were mentioned.Message ID: <JonathanTreffler/backblaze-personal-wine-container/issues/127/1959897962@ github.com>

traktuner commented 6 months ago

Backblaze version 9 was released in September 2023. We released version 1.6 of our container with auto-update functionality mid January 2024 which was a highly requested feature. So all who started with the container before September 2023 were on Backblaze v8 until mid January 2024 and then got auto-updated to Backblaze version 9.

Maybe v8 can be found at archive.org - but I am not sure if downgrading a major version is supported at all. So you may break your Backblaze installation. I would not suggest to try that, and we don't support that in any way. A better solution would be to wait for the next Backblaze update and see if that fixes your speed issue.

I did a speedtest inside my Backblaze docker container (latest image / ubuntu20 / main) and the upload speed matches my internet speed (100mbit/s) and the Backblaze backups saturate my upload bandwith. I am pretty sure that this has nothing to do with our docker container implementation - the wine translation layer should not be a big bottleneck.

Unfortunately I have no way in reproducing the slow backup issue and the current stable release works as expected.

I will convert this issue into a discussion since this fits better in my opinion.