search

rclone / rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
https://rclone.org
MIT License
45.98k stars 4.11k forks source link

Request: support B2’s (beta) server-side copy #3210

Closed Jwink3101 closed 5 years ago

Jwink3101 commented 5 years ago

B2 now has a beta server-side copy. See the docs and a relevant Reddit post

I am very excited for this!

ncw commented 5 years ago

Thanks for the nudge :-)

This should hopefully be pretty straight forward to implement it.

Depending on testing I may/may not leave it out of the 1.48 release!

chiraag-nataraj commented 5 years ago

I'd be happy to help test if that means it makes it in :wink:

ncw commented 5 years ago

OK here is a first attempt.

I've given it a light testing and it passes the integration tests.

Please test. Can you try copying big files too and see how that works? How long does copying a 1GB file take? Renaming things in a mount should work now too but I haven't tested that.

https://beta.rclone.org/branch/v1.47.0-075-gd70d9484-fix-3210-b2-copy-beta/ (uploaded in 15-30 mins)

chiraag-nataraj commented 5 years ago

I'm on Debian, so I'll have to compile it. Never compiled a go program before, but I'll figure it out :joy:

jikamens commented 5 years ago

The changes here are great, but they don't take full advantage of the new functionality. In particular, the new functionality includes partial file copies, which means it should be possible to use it to do incremental file updates. This is actually a really big potential win for backups (though, alas, not for encrypted backups :-/ ).

ncw commented 5 years ago

I'm on Debian, so I'll have to compile it. Never compiled a go program before, but I'll figure it out

The build went wrong... I've restarted it so linux builds should appear in 15-30 mins hopefully.

ncw commented 5 years ago

@jikamens wrote:

The changes here are great, but they don't take full advantage of the new functionality. In particular, the new functionality includes partial file copies, which means it should be possible to use it to do incremental file updates. This is actually a really big potential win for backups (though, alas, not for encrypted backups :-/ ).

I don't think the new functionality includes updating existing files - can you point to some docs which say that?

jikamens commented 5 years ago

Backblaze claims that the new functionality can be used for incremental backups, but like you, after reading the API I can't figure out how that would work. I've reached out to them to ask for clarification.

ncw commented 5 years ago

Assuming you can use the copy interface to copy a file over itself while updating the metadata (this is how s3 works) then I can also make b2 support setting modification times on files which have already been uploaded :-)

jikamens commented 5 years ago

Yes, you can supposedly do that. But so far they haven't responded with a concrete explanation of their claim that incremental backups are possible. It only seems possible if what they're talking about is storing a file in multiple pieces in the underlying storage back-end and then having a database indicating how to reassemble pieces into a file in order to restore it, and rclone doesn't do that, at least not for the B2 storage back-end.

chiraag-nataraj commented 5 years ago

Please test. Can you try copying big files too and see how that works? How long does copying a 1GB file take? Renaming things in a mount should work now too but I haven't tested that.

I tried renaming stuff in a mount and it worked (the old file was hidden and the new file was created two seconds before the old one was hidden, indicating that the file wasn't uploaded againt - it's actually the rclone deb package haha)! I'll try copying large files later and report back (although I haven't actually ever really tried uploading super-large files, so I guess we'll see how that goes...).

Already lightly tested server-side copy for a smaller file (compared with the release version to figure out the baseline) and it's great :D

jikamens commented 5 years ago

sigh Confirmation that they expect reassembly to be implemented in the backup software: https://www.backblaze.com/blog/backblaze-b2-copy-file-beta-is-now-public/#comment-4471290462

dquintela commented 5 years ago

I am testing also tonight on Windows..

jikamens commented 5 years ago

Wait a minute, it might be possible after all!

https://disqus.com/home/discussion/backblaze/backblaze_b2_copy_file_beta_is_now_public/?utm_source=reply&utm_medium=email&utm_content=comment_date#comment-4471347501

dquintela commented 5 years ago

I am testing also tonight on Windows..

Using rclone mount on windows $> rclone -v mount b2-crypt:/ X: --fast-list -P --volname "B2-CLOUD-CRYPT" --log-file "c:\Local\cloud\rclone-data\log.log" --cache-dir "c:\Local\cloud\rclone-data\cache-vfs" --vfs-cache-mode writes --config "c:\Local\cloud\rclone.conf"

I am having problems renaming files (First attempt has failed and then seem to worked out nssm-2.24 - Copy2.zip -> nssm-2.24 - Copy3.zip) The last attemp was after created an empty folder, move the file there - also failed. "nssm-2.24 - Copy3.zip" -> "New Folder\nssm-2.24 - Copy3.zip"

Logs snippet.. 2019/05/22 01:32:57 INFO : b2-caching: Cache DB path: c:\Local\cloud\rclone-data\cache-backend\b2-caching.db 2019/05/22 01:32:57 INFO : b2-caching: Cache chunk path: c:\Local\cloud\rclone-data\cache-backend-chunks\b2-caching 2019/05/22 01:32:57 INFO : b2-caching: Chunk Memory: true 2019/05/22 01:32:57 INFO : b2-caching: Chunk Size: 10M 2019/05/22 01:32:57 INFO : b2-caching: Chunk Total Size: 10G 2019/05/22 01:32:57 INFO : b2-caching: Chunk Clean Interval: 1m0s 2019/05/22 01:32:57 INFO : b2-caching: Workers: 16 2019/05/22 01:32:57 INFO : b2-caching: File Age: 1h0m0s 2019/05/22 01:32:57 INFO : b2-caching: Cache DB path: c:\Local\cloud\rclone-data\cache-backend\b2-caching.db 2019/05/22 01:32:57 INFO : b2-caching: Cache chunk path: c:\Local\cloud\rclone-data\cache-backend-chunks\b2-caching 2019/05/22 01:32:57 INFO : b2-caching: Chunk Memory: true 2019/05/22 01:32:57 INFO : b2-caching: Chunk Size: 10M 2019/05/22 01:32:57 INFO : b2-caching: Chunk Total Size: 10G 2019/05/22 01:32:57 INFO : b2-caching: Chunk Clean Interval: 1m0s 2019/05/22 01:32:57 INFO : b2-caching: Workers: 16 2019/05/22 01:32:57 INFO : b2-caching: File Age: 1h0m0s 2019/05/22 01:32:57 INFO : Encrypted drive 'b2-crypt:/': poll-interval is not supported by this remote 2019/05/22 01:33:00 NOTICE: nssm-2.24 - Copy.zip: Encrypts to "aaaaaaa" 2019/05/22 01:33:00 NOTICE: yubikey-manager-qt-1.0.1-win64.exe: Encrypts to "bbbbbbbbbbbb" 2019/05/22 01:33:00 NOTICE: Artigo MP Revista Sociologia: Encrypts to "cccccccccccc" 2019/05/22 01:33:00 NOTICE: nssm-2.24.zip: Encrypts to "dddddddddd" 2019/05/22 01:33:00 NOTICE: anabela0.xls: Encrypts to "eeeeeeeeeee" 2019/05/22 01:33:00 NOTICE: winfsp-1.4.19049.msi: Encrypts to "ffffffffff" 2019/05/22 01:33:00 NOTICE: APRESENTACAO PROVAS.pptx: Encrypts to "gggggggggggggg" 2019/05/22 01:33:00 NOTICE: Tese Denise Henriques Quintela - Doutoramento Politicas Publicas.pdf: Encrypts to "hhhhhhhhhhhhh" 2019/05/22 01:33:00 NOTICE: TESE: Encrypts to "iiiiiiiiiiii" 2019/05/22 01:33:00 NOTICE: anabela.xls: Encrypts to "jjjjjjjjjjjjj" 2019/05/22 01:33:00 NOTICE: sqldeveloper-18.3.0.277.2354-no-jre.zip: Encrypts to "kkkkkkkkkkkk" 2019/05/22 01:33:00 NOTICE: rclone-v1.46-windows-amd64.zip: Encrypts to "lllllllllllllllll" 2019/05/22 01:33:00 NOTICE: CV: Encrypts to "mmmmmmmmmmmmm" 2019/05/22 01:33:00 NOTICE: yubioath-desktop-4.3.5-win64.exe: Encrypts to "nnnnnnnnnnnn" 2019/05/22 01:33:22 ERROR : aaaaaaa: error moving in cache: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:22 ERROR : nssm-2.24 - Copy.zip: Failed to copy: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:22 ERROR : nssm-2.24 - Copy.zip: Not deleting source as copy failed: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:22 ERROR : nssm-2.24 - Copy.zip: File.Rename error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:22 ERROR : nssm-2.24 - Copy.zip: Dir.Rename error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:22 ERROR : IO error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:27 ERROR : aaaaaaa: error moving in cache: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:27 ERROR : nssm-2.24 - Copy.zip: Failed to copy: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:27 ERROR : nssm-2.24 - Copy.zip: Not deleting source as copy failed: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:27 ERROR : nssm-2.24 - Copy.zip: File.Rename error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:27 ERROR : nssm-2.24 - Copy.zip: Dir.Rename error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:27 ERROR : IO error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:33:35 INFO : : copy: cache expired 2019/05/22 01:33:35 INFO : : copy: cache expired 2019/05/22 01:33:35 INFO : nssm-2.24 - Copy.zip: Copied (server side copy) 2019/05/22 01:33:35 INFO : nssm-2.24 - Copy.zip: Deleted 2019/05/22 01:35:45 INFO : b2-caching: Cache DB path: c:\Local\cloud\rclone-data\cache-backend\b2-caching.db 2019/05/22 01:35:45 INFO : b2-caching: Cache chunk path: c:\Local\cloud\rclone-data\cache-backend-chunks\b2-caching 2019/05/22 01:35:45 INFO : b2-caching: Chunk Memory: true 2019/05/22 01:35:45 INFO : b2-caching: Chunk Size: 10M 2019/05/22 01:35:45 INFO : b2-caching: Chunk Total Size: 10G 2019/05/22 01:35:45 INFO : b2-caching: Chunk Clean Interval: 1m0s 2019/05/22 01:35:45 INFO : b2-caching: Workers: 16 2019/05/22 01:35:45 INFO : b2-caching: File Age: 1h0m0s 2019/05/22 01:35:45 INFO : b2-caching: Cache DB path: c:\Local\cloud\rclone-data\cache-backend\b2-caching.db 2019/05/22 01:35:45 INFO : b2-caching: Cache chunk path: c:\Local\cloud\rclone-data\cache-backend-chunks\b2-caching 2019/05/22 01:35:45 INFO : b2-caching: Chunk Memory: true 2019/05/22 01:35:45 INFO : b2-caching: Chunk Size: 10M 2019/05/22 01:35:45 INFO : b2-caching: Chunk Total Size: 10G 2019/05/22 01:35:45 INFO : b2-caching: Chunk Clean Interval: 1m0s 2019/05/22 01:35:45 INFO : b2-caching: Workers: 16 2019/05/22 01:35:45 INFO : b2-caching: File Age: 1h0m0s 2019/05/22 01:35:45 INFO : Encrypted drive 'b2-crypt:/': poll-interval is not supported by this remote 2019/05/22 01:35:50 NOTICE: yubikey-manager-qt-1.0.1-win64.exe: Encrypts to "bbbbbbbbbbbb" 2019/05/22 01:35:50 NOTICE: Artigo MP Revista Sociologia: Encrypts to "cccccccccccc" 2019/05/22 01:35:50 NOTICE: nssm-2.24 - Copy2.zip: Encrypts to "ooooooooo" 2019/05/22 01:35:50 NOTICE: nssm-2.24.zip: Encrypts to "dddddddddd" 2019/05/22 01:35:50 NOTICE: anabela0.xls: Encrypts to "eeeeeeeeeee" 2019/05/22 01:35:50 NOTICE: winfsp-1.4.19049.msi: Encrypts to "ffffffffff" 2019/05/22 01:35:50 NOTICE: APRESENTACAO PROVAS.pptx: Encrypts to "gggggggggggggg" 2019/05/22 01:35:50 NOTICE: Tese Denise Henriques Quintela - Doutoramento Politicas Publicas.pdf: Encrypts to "hhhhhhhhhhhhh" 2019/05/22 01:35:50 NOTICE: TESE: Encrypts to "iiiiiiiiiiii" 2019/05/22 01:35:50 NOTICE: anabela.xls: Encrypts to "jjjjjjjjjjjjj" 2019/05/22 01:35:50 NOTICE: sqldeveloper-18.3.0.277.2354-no-jre.zip: Encrypts to "kkkkkkkkkkkk" 2019/05/22 01:35:50 NOTICE: rclone-v1.46-windows-amd64.zip: Encrypts to "ppppppp" 2019/05/22 01:35:50 NOTICE: CV: Encrypts to "qqqqqqqqqqqq" 2019/05/22 01:35:50 NOTICE: yubioath-desktop-4.3.5-win64.exe: Encrypts to "ssssssss" 2019/05/22 01:36:03 INFO : : copy: cache expired 2019/05/22 01:36:03 INFO : : copy: cache expired 2019/05/22 01:36:03 INFO : nssm-2.24 - Copy2.zip: Copied (server side copy) 2019/05/22 01:36:03 INFO : nssm-2.24 - Copy2.zip: Deleted 2019/05/22 01:36:15 INFO : : mkdir: cache expired 2019/05/22 01:36:46 ERROR : rrrrrrrrrrr: error moving in cache: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:36:46 ERROR : nssm-2.24 - Copy3.zip: Failed to copy: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:36:46 ERROR : nssm-2.24 - Copy3.zip: Not deleting source as copy failed: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:36:46 ERROR : nssm-2.24 - Copy3.zip: File.Rename error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:36:46 ERROR : nssm-2.24 - Copy3.zip: Dir.Rename error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:36:46 ERROR : IO error: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed) 2019/05/22 01:44:28 NOTICE: yubikey-manager-qt-1.0.1-win64.exe: Encrypts to "bbbbbbbbbbbb" 2019/05/22 01:44:28 NOTICE: Artigo MP Revista Sociologia: Encrypts to "cccccccccccc" 2019/05/22 01:44:28 NOTICE: nssm-2.24.zip: Encrypts to "dddddddddd" 2019/05/22 01:44:28 NOTICE: anabela0.xls: Encrypts to "eeeeeeeeeee" 2019/05/22 01:44:28 NOTICE: winfsp-1.4.19049.msi: Encrypts to "ffffffffff" 2019/05/22 01:44:28 NOTICE: APRESENTACAO PROVAS.pptx: Encrypts to "gggggggggggggg" 2019/05/22 01:44:28 NOTICE: Tese Denise Henriques Quintela - Doutoramento Politicas Publicas.pdf: Encrypts to "hhhhhhhhhhhhh" 2019/05/22 01:44:28 NOTICE: TESE: Encrypts to "iiiiiiiiiiii" 2019/05/22 01:44:28 NOTICE: anabela.xls: Encrypts to "jjjjjjjjjjjjj" 2019/05/22 01:44:28 NOTICE: sqldeveloper-18.3.0.277.2354-no-jre.zip: Encrypts to "kkkkkkkkkkkk" 2019/05/22 01:44:28 NOTICE: rclone-v1.46-windows-amd64.zip: Encrypts to "ppppppp" 2019/05/22 01:44:28 NOTICE: CV: Encrypts to "qqqqqqqqqqqq" 2019/05/22 01:44:28 NOTICE: nssm-2.24 - Copy3.zip: Encrypts to "rrrrrrrrrrr" 2019/05/22 01:44:28 NOTICE: yubioath-desktop-4.3.5-win64.exe: Encrypts to "ssssssss"

rclone.conf [b2] type = b2 account = xxxxxxxxxxxxxxxxxxxxx key = xxxxxxxxxxxxxxxxxxxxx hard_delete = true upload_cutoff = 100M chunk_size = 96M

[b2-caching] type = cache remote = b2:dquintela-rclone chunk_size = 10M info_age = 1h0m0s chunk_total_size = 10G db_path = c:\Local\cloud\rclone-data\cache-backend chunk_path = c:\Local\cloud\rclone-data\cache-backend-chunks db_purge = false read_retries = 10 workers = 16 writes = false

[b2-crypt] type = crypt remote = b2-caching: filename_encryption = standard directory_name_encryption = true password = xxxxxxxxxxxxxxxxxxxxx password2 = xxxxxxxxxxxxxxxxxxxxx show_mapping = true

dquintela commented 5 years ago

Some note on previous; "java.io.IOException" on go ? or is it an server side error ?

dquintela commented 5 years ago

Just restarted mount to increase log verbosity.. Single file rename OK "Single File" -> "New Folder\Single File" OK.

Some side effect or was indeed some server errors..

dquintela commented 5 years ago

Beware: The moved files in my configuration seem to end with 0 bytes; Confirmed on backblaze.

xxxxxx 32.0 bytes 05/22/2019 02:11 yyyyyyy 32.0 bytes 05/22/2019 02:06

ncw commented 5 years ago

2019/05/22 01:33:22 ERROR : aaaaaaa: error moving in cache: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed)

That error must have come from backblaze - rclone is written in Go not java.

Backblaze did say it was a beta!

If you can reproduce that then we should probably report it to them. If you run with -vv --dump bodies it will show the full HTTP transaction.

Beware: The moved files in my configuration seem to end with 0 bytes; Confirmed on backblaze.

xxxxxx 32.0 bytes 05/22/2019 02:11 yyyyyyy 32.0 bytes 05/22/2019 02:06

Are they appearing as 32.0 bytes? That is odd!

dquintela commented 5 years ago

2019/05/22 01:33:22 ERROR : aaaaaaa: error moving in cache: failed to create inputStream from source: java.io.IOException: input stream already created (999 read_failed)

That error must have come from backblaze - rclone is written in Go not java.

Backblaze did say it was a beta!

If you can reproduce that then we should probably report it to them. If you run with -vv --dump bodies it will show the full HTTP transaction.

Beware: The moved files in my configuration seem to end with 0 bytes; Confirmed on backblaze. xxxxxx 32.0 bytes 05/22/2019 02:11 yyyyyyy 32.0 bytes 05/22/2019 02:06

Are they appearing as 32.0 bytes? That is odd!

I suppose that may be a side effect of crypt backend ?

dquintela commented 5 years ago

If you can reproduce that then we should probably report it to them. If you run with -vv --dump bodies it will show the full HTTP transaction.

Tonight, I'll make more tests with that flag active!

ncw commented 5 years ago

I suppose that may be a side effect of crypt backend ?

Where did you see that - on the backblaze website? I think it is probably a bug at their end.

Tonight, I'll make more tests with that flag active!

Great

dquintela commented 5 years ago

Where did you see that - on the backblaze website? I think it is probably a bug at their end.

That listing with 32 bytes was the bucket browser on their site..

I'll try to make 3 tests, one direct to b2, other with caching and finally with crypt on top, running the bodies and see if I can trigger the issues again.

nilayp commented 5 years ago

Nilay from Backblaze here. @dquintela- can you reliably reproduce this issue? We think we might know what's happening and are planning to deploy a fix, but have been unable to reproduce the problem, so can't 100% validate that it will fix the issue.

Let me know if you can validate the issue. Just respond here or b2beta@backblaze.com.

dquintela commented 5 years ago

Nilay from Backblaze here. @dquintela- can you reliably reproduce this issue? We think we might know what's happening and are planning to deploy a fix, but have been unable to reproduce the problem, so can't 100% validate that it will fix the issue.

Let me know if you can validate the issue. Just respond here or b2beta@backblaze.com.

Yesterday I spent several ours testing. I replicated the problem, but it was somewhat intermittent - at least, failed to dig a root cause.

I'll retry again today.

What I got so far, mostly for @ncw , because I was making my tests through the command line and "rclone mount", was "MKDIR DIR" failing because of rclone not creating the empty directories placeholders, likely related to #100, #1766, #1837, #2505 and so on..

dquintela commented 5 years ago

"MKDIR DIR" failing because

Yet, in "Windows Explorer" is working - we see windows creating a hidden file on created directory, seems ".desktop"

This behaviour seems to be impacting caching behaviour also...

So you see, had several problems :)

nilayp commented 5 years ago

Ok, thanks for the update. If you see another Java exception, please let us know. We believe the problem has been fixed in production.

ncw commented 5 years ago

Ok, thanks for the update. If you see another Java exception, please let us know. We believe the problem has been fixed in production.

Thank you @nilayp :-) It is coming up for an rclone release in a few days. Do you think I should release this feature in rclone then or do you think it needs some more time to mature at backblaze?

The feature can always be disabled with --disable copy.

ncw commented 5 years ago

I've implemented SetModTime now too using server side copy.

This creates a new version using a server side copy while updating the metadata.

The means that times can be updated in syncing and that rclone touch now works with b2.

https://beta.rclone.org/branch/v1.47.0-077-g073d3bb0-fix-3210-b2-copy-beta/ (uploaded in 15-30 mins)

ncw commented 5 years ago

SetModtime will be creating a new version of the file.

Should rclone delete the old version automatically? Thoughts?

dquintela commented 5 years ago

Should rclone delete the old version automatically? Thoughts?

My opinion is yes (this shouldn't be influenced by "hard_delete = true" - I would expect that to be only for files changes by the user?)

jsc802 commented 5 years ago

Hi, I am new to this, can someone tell me what command to use to test this (renaminga file)?

ncw commented 5 years ago

Hi, I am new to this, can someone tell me what command to use to test this (renaminga file)?

To rename a single file or a directory

rclone moveto b2:bucket/path/to/file b2:bucket/path/to/new_file_name
sedlund commented 5 years ago

@ncw

https://beta.rclone.org/branch/v1.47.0-077-g073d3bb0-fix-3210-b2-copy-beta/ (uploaded in 15-30 mins)

Seems only windows and osx got built - would like to try this on linux

ncw commented 5 years ago

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.48

AndrewPaglusch commented 5 years ago

I'm running into an issue with the latest beta when trying to do a server-side copy via the mount command. It looks like copying a zero-byte file fails. Copying larger files works as expected.

Please let me know if I can provide any more information that may be helpful.

root@pve01:~# touch b2/file
root@pve01:~# cp b2/file b2/newfile
cp: error reading 'b2/file': Input/output error
root@pve01:~# rclone mount -vvv backblazeb2:mybucket b2
2019/06/14 15:11:27 DEBUG : rclone: Version "v1.47.0-128-g22368b99-beta" starting with parameters ["rclone" "mount" "-vvv" "backblazeb2:mybucket" "b2"]
2019/06/14 15:11:27 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2019/06/14 15:11:28 DEBUG : B2 bucket mybucket: Mounting on "b2"
2019/06/14 15:11:28 INFO  : B2 bucket mybucket: poll-interval is not supported by this remote
2019/06/14 15:11:28 DEBUG : Adding path "vfs/forget" to remote control registry
2019/06/14 15:11:28 DEBUG : Adding path "vfs/refresh" to remote control registry
2019/06/14 15:11:28 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2019/06/14 15:11:28 DEBUG : : Root:
2019/06/14 15:11:28 DEBUG : : >Root: node=/, err=<nil>
2019/06/14 15:11:32 DEBUG : /: Attr:
2019/06/14 15:11:32 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2019/06/14 15:11:34 DEBUG : /: Lookup: name="file"
2019/06/14 15:11:34 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2019/06/14 15:11:34 DEBUG : /: Create: name="file"
2019/06/14 15:11:34 DEBUG : file: Open: flags=O_WRONLY|O_CREATE|0x800
2019/06/14 15:11:34 DEBUG : file: >Open: fd=file (w), err=<nil>
2019/06/14 15:11:34 DEBUG : /: >Create: node=file, handle=&{file (w)}, err=<nil>
2019/06/14 15:11:34 DEBUG : file: Attr:
2019/06/14 15:11:34 DEBUG : file: >Attr: a=valid=1s ino=0 size=0 mode=-rw-r--r--, err=<nil>
2019/06/14 15:11:34 DEBUG : &{file (w)}: Flush:
2019/06/14 15:11:34 DEBUG : file: WriteFileHandle.Flush unwritten handle, writing 0 bytes to avoid race conditions
2019/06/14 15:11:34 DEBUG : &{file (w)}: >Flush: err=<nil>
2019/06/14 15:11:34 DEBUG : file: Setattr: a=Setattr [ID=0x6 Node=0x2 Uid=0 Gid=0 Pid=27966] atime=2019-06-14 15:11:34.737146578 -0500 CDT atime=now mtime=2019-06-14 15:11:34.737146578 -0500 CDT mtime=now handle=INVALID-0x0
2019/06/14 15:11:34 DEBUG : file: >Setattr: err=<nil>
2019/06/14 15:11:34 DEBUG : file: Attr:
2019/06/14 15:11:34 DEBUG : file: >Attr: a=valid=1s ino=0 size=0 mode=-rw-r--r--, err=<nil>
2019/06/14 15:11:34 DEBUG : &{file (w)}: Flush:
2019/06/14 15:11:34 DEBUG : B2 bucket mybucket: File to upload is small (0 bytes), uploading instead of streaming
2019/06/14 15:11:35 INFO  : file: Copied (new)
2019/06/14 15:11:36 DEBUG : file: File.applyPendingModTime OK
2019/06/14 15:11:36 DEBUG : &{file (w)}: >Flush: err=<nil>
2019/06/14 15:11:36 DEBUG : &{file (w)}: Release:
2019/06/14 15:11:36 DEBUG : file: WriteFileHandle.Release nothing to do
2019/06/14 15:11:36 DEBUG : &{file (w)}: >Release: err=<nil>
2019/06/14 15:11:39 DEBUG : /: Attr:
2019/06/14 15:11:39 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2019/06/14 15:11:40 DEBUG : /: ReadDirAll:
2019/06/14 15:11:40 DEBUG : /: >ReadDirAll: item=1, err=<nil>
2019/06/14 15:11:40 DEBUG : file: Attr:
2019/06/14 15:11:40 DEBUG : file: >Attr: a=valid=1s ino=0 size=0 mode=-rw-r--r--, err=<nil>
2019/06/14 15:11:40 DEBUG : /: Attr:
2019/06/14 15:11:40 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2019/06/14 15:11:40 DEBUG : /: ReadDirAll:
2019/06/14 15:11:40 DEBUG : /: >ReadDirAll: item=1, err=<nil>
2019/06/14 15:11:40 DEBUG : /: Attr:
2019/06/14 15:11:40 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2019/06/14 15:11:42 DEBUG : /: Attr:
2019/06/14 15:11:42 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2019/06/14 15:11:45 DEBUG : /: Lookup: name="newfile"
2019/06/14 15:11:45 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2019/06/14 15:11:45 DEBUG : file: Attr:
2019/06/14 15:11:45 DEBUG : file: >Attr: a=valid=1s ino=0 size=0 mode=-rw-r--r--, err=<nil>
2019/06/14 15:11:45 DEBUG : /: Lookup: name="newfile"
2019/06/14 15:11:45 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2019/06/14 15:11:45 DEBUG : file: Open: flags=OpenReadOnly
2019/06/14 15:11:45 DEBUG : file: Open: flags=O_RDONLY
2019/06/14 15:11:45 DEBUG : file: >Open: fd=file (r), err=<nil>
2019/06/14 15:11:45 DEBUG : file: >Open: fh=&{file (r)}, err=<nil>
2019/06/14 15:11:45 DEBUG : /: Lookup: name="newfile"
2019/06/14 15:11:45 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2019/06/14 15:11:45 DEBUG : /: Create: name="newfile"
2019/06/14 15:11:45 DEBUG : newfile: Open: flags=O_WRONLY|O_CREATE|O_EXCL
2019/06/14 15:11:45 DEBUG : newfile: >Open: fd=newfile (w), err=<nil>
2019/06/14 15:11:45 DEBUG : /: >Create: node=newfile, handle=&{newfile (w)}, err=<nil>
2019/06/14 15:11:45 DEBUG : newfile: Attr:
2019/06/14 15:11:45 DEBUG : newfile: >Attr: a=valid=1s ino=0 size=0 mode=-rw-r--r--, err=<nil>
2019/06/14 15:11:45 DEBUG : &{file (r)}: Read: len=4096, offset=0
2019/06/14 15:11:45 DEBUG : file: ChunkedReader.openRange at 0 length 134217728
2019/06/14 15:11:45 DEBUG : &{file (r)}: >Read: read=0, err=failed to open for download: byte range bytes=0-134217727 for a file of length 0 (416 range_not_satisfiable)
2019/06/14 15:11:45 DEBUG : &{newfile (w)}: Flush:
2019/06/14 15:11:45 DEBUG : newfile: WriteFileHandle.Flush unwritten handle, writing 0 bytes to avoid race conditions
2019/06/14 15:11:45 DEBUG : &{newfile (w)}: >Flush: err=<nil>
2019/06/14 15:11:45 DEBUG : &{newfile (w)}: Release:
2019/06/14 15:11:45 DEBUG : newfile: WriteFileHandle.Release closing
2019/06/14 15:11:45 DEBUG : B2 bucket mybucket: File to upload is small (0 bytes), uploading instead of streaming
2019/06/14 15:11:45 DEBUG : &{file (r)}: Flush:
2019/06/14 15:11:45 DEBUG : &{file (r)}: >Flush: err=<nil>
2019/06/14 15:11:45 DEBUG : &{file (r)}: Release:
2019/06/14 15:11:45 DEBUG : &{file (r)}: >Release: err=<nil>
2019/06/14 15:11:51 INFO  : newfile: Copied (new)
2019/06/14 15:11:51 DEBUG : &{newfile (w)}: >Release: err=<nil>
ncw commented 5 years ago

I'm running into an issue with the latest beta when trying to do a server-side copy via the mount command. It looks like copying a zero-byte file fails. Copying larger files works as expected.

Please let me know if I can provide any more information that may be helpful.

root@pve01:~# touch b2/file
root@pve01:~# cp b2/file b2/newfile
cp: error reading 'b2/file': Input/output error

That doesn't do a server side copy. mv b2/file b2/newfile does and that does work in my tests.

So this must be something else...

It looks like the problem is reading the 0 length file on the mount

$ cat /mnt/tmp/empty2.txt 
cat: /mnt/tmp/empty2.txt: Input/output error

Which is caused by

2019/06/15 09:47:07 DEBUG : &{empty2.txt (r)}: >Read: read=0, err=failed to open for download: byte range bytes=0-134217727 for a file of length 0 (416 range_not_satisfiable)

@AndrewPaglusch - can you put this in a new issue? Thanks!