rClone fails when configured with `download_url` from cloudflare-b2 deployed worker domain

[Azathothas]

Note: I am not a dev, so some of the technical information may not be accurate

• My wrangler.toml

  name = "b2-mybucket"
  workers_dev = true
  compatibility_date = "2023-09-04"
  routes = [
  { pattern = "b2.my-domain.com", custom_domain = true, zone_name = "my-domain.com" }
  ]
  main = "index.js"
  [vars]
  # Bucket Key : capabilities: bypassGovernance, deleteBuckets, deleteFiles, deleteKeys, listBuckets, listFiles, listKeys, readBucketEncryption, readBucketReplications, readBucketRetentions, readBuckets, readFileLegalHolds, readFileRetentions, readFiles, shareFiles, writeBucketEncryption, writeBucketReplications, writeBucketRetentions, writeBuckets, writeFileLegalHolds, writeFileRetentions, writeFiles, writeKeys
  B2_APPLICATION_KEY_ID = "00redacted00000000001"
  #B2_APPLICATION_KEY = (set via: echo "key" | wrangler secret put "B2_APPLICATION_KEY")
  B2_ENDPOINT = "s3.us-east-005.backblazeb2.com"
  BUCKET_NAME = "mybucket"
  ALLOW_LIST_BUCKET = "false"

• My Bucket Configuration

  

• My rclone.conf

  [b2]
  type = b2
  account = 00redacted00000000001
  key = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  download_url = https://b2.my-domain.com
  hard_delete = true
  download_auth_duration = 1w

• Downloading https://b2.my-domain.com/my-file Works ✅

• Using rclone copy|copyto Fails ❌

  
  #Error when I try copying from Local system to b2 bucket
  ❯ rclone copyto  "$LOCAL_FILESYSTEM/my-file" "b2:/my-bucket/my-file"

Failed to copyto: failed to HEAD for download: Unknown 403 Forbidden (403 unknown)

Same error if I try copying from bucket to my local system

❯ rclone copyto "b2:/my-bucket/my-file" "$LOCAL_FILESYSTEM/my-file"

Failed to copyto: failed to HEAD for download: Unknown 403 Forbidden (403 unknown)

- Using `--no-check-dest` will work **but exit with same error** ❌
```bash
❯ rclone copyto  "./my-file" "b2:/my-bucket/my-file" --no-check-dest
2024/03/10 19:00:33 ERROR : my-file: Failed to copy: multi-thread copy: failed to find object after copy: failed to HEAD for download: Unknown 403 Forbidden (403 unknown)
2024/03/10 19:00:33 ERROR : Fatal error received - not attempting retries
Transferred:        2.967 GiB / 2.967 GiB, 100%, 185.240 KiB/s, ETA 0s
Errors:                 1 (fatal error encountered)
Elapsed time:     12m48.5s
2024/03/10 19:00:33 Failed to copyto: multi-thread copy: failed to find object after copy: failed to HEAD for download: Unknown 403 Forbidden (403 unknown)

• rclone mount

  • Using File Explorer/CLI to Copy from local-folder to b2:/my-bucket-rclone-mount Works ✅

• Using File Explorer/CLI to Copy from b2:/my-bucket-rclone-mount to local-folder Fails ❌

  The request could not be performed because of an I/O device error.

---

• Possible Cause

As per https://rclone.org/b2/#b2-download-url

Custom endpoint for downloads. This is usually set to a Cloudflare CDN URL as Backblaze offers free egress for data downloaded through the Cloudflare network. Rclone works with private buckets by sending an "Authorization" header. If the custom endpoint rewrites the requests for authentication, e.g., in Cloudflare Workers, this header needs to be handled properly. Leave blank if you want to use the endpoint provided by Backblaze.

May also be related:

@sshockwave filed a similar issue: https://forum.rclone.org/t/backblaze-b2-with-cloudflare-worker-conflicts-on-the-authorization/21353/3

https://github.com/rclone/rclone/pull/4896

---

As soon as I remove download_url = https://b2.my-domain.com from rclone.conf , everything works as expected, but then there would be no point in using BackBlaze at all since egress cost alone would be too expensive for me.

[kapitanluffy]

Fixed it by patching the code itself

[Azathothas]

Hi @kapitanluffy , thank you for sending a fix. I had forgotten that I had even created this issue. Unfortunately, I went with another provider, so I can't test if this works.

[kapitanluffy]

Hi @kapitanluffy , thank you for sending a fix. I had forgotten that I had even created this issue. Unfortunately, I went with another provider, so I can't test if this works.

No worries. I ran into a similar issue so I thought I shouldnshare it here if someone sees this.

[metadaddy]

The issue here is that as explained in the docs, rclone assumes that it needs to prefix the path with file/, but this worker is using the S3-compatible API to access B2, which doesn't have that file/ prefix.

I'll look at fixing this by introducing a new environment variable.