JuiceFS WebDAV backend does not work with Rclone WebDAV server in Windows

[stevefan1999-personal]

What happened:

I can't use JuiceFS with WebDAV served by rclone just as the title indicates

What you expected to happen:

It works

How to reproduce it (as minimally and precisely as possible):

1. Install WinFSP
2. Install Rclone and JuiceFS
3. Setup any rclone remote
4. Serve a rclone remote using WebDAV
5. (Optional) Copy a random file to the WebDAV "drive" using any client (mine is using native Windows network drive in explorer) and confirm writing works
6. Format a WebDAV backed JuiceFS using any metadata engine of your likes. I use Sqlite so that it is the original environment plus it is easier to create
7. Mount the WebDAV backed JuiceFS to any drive
8. Try to copy a file to the JuiceFS drive
9. Expect Failure
10. (Optional) Repeat step 6 but use a local filestore and compare the result

Anything else we need to know?

It seems like something is wrong with rclone more than juicefs?

2022/02/17 20:39:55 DEBUG : /chunks/0/4/4001_0_4194304: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:39:55 DEBUG : /chunks/0/4/4001_0_4194304: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:39:55 INFO  : /chunks/0/4/4001_0_4194304: PUT from 127.0.0.1:58578
2022/02/17 20:39:55 DEBUG : /chunks/0/4/4001_1_2570038: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:39:55 DEBUG : /chunks/0/4/4001_1_2570038: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:39:55 INFO  : /chunks/0/4/4001_1_2570038: PUT from 127.0.0.1:58579
2022/02/17 20:39:58 DEBUG : /chunks/0/4/4001_0_4194304: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:39:58 DEBUG : /chunks/0/4/4001_0_4194304: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:39:58 INFO  : /chunks/0/4/4001_0_4194304: PUT from 127.0.0.1:58586
2022/02/17 20:39:58 DEBUG : /chunks/0/4/4001_1_2570038: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:39:58 DEBUG : /chunks/0/4/4001_1_2570038: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:39:58 INFO  : /chunks/0/4/4001_1_2570038: PUT from 127.0.0.1:58587
2022/02/17 20:40:02 DEBUG : /chunks/0/4/4001_0_4194304: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:40:02 DEBUG : /chunks/0/4/4001_0_4194304: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:40:02 INFO  : /chunks/0/4/4001_0_4194304: PUT from 127.0.0.1:58592
2022/02/17 20:40:04 DEBUG : /chunks/0/4/4001_1_2570038: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:40:04 DEBUG : /chunks/0/4/4001_1_2570038: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:40:04 INFO  : /chunks/0/4/4001_1_2570038: PUT from 127.0.0.1:58596
2022/02/17 20:40:11 DEBUG : /chunks/0/4/4001_0_4194304: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:40:11 DEBUG : /chunks/0/4/4001_0_4194304: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:40:11 INFO  : /chunks/0/4/4001_0_4194304: PUT from 127.0.0.1:58605
2022/02/17 20:40:15 DEBUG : /chunks/0/4/4001_1_2570038: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:40:15 DEBUG : /chunks/0/4/4001_1_2570038: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:40:15 INFO  : /chunks/0/4/4001_1_2570038: PUT from 127.0.0.1:58611
2022/02/17 20:40:29 DEBUG : /chunks/0/4/4001_0_4194304: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:40:29 DEBUG : /chunks/0/4/4001_0_4194304: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:40:29 INFO  : /chunks/0/4/4001_0_4194304: PUT from 127.0.0.1:58666
2022/02/17 20:40:33 DEBUG : /chunks/0/4/4001_1_2570038: OpenFile: flags=O_RDWR|O_CREATE|O_TRUNC, perm=-rw-rw-rw-
2022/02/17 20:40:33 DEBUG : /chunks/0/4/4001_1_2570038: >OpenFile: fd=<nil>, err=file does not exist
2022/02/17 20:40:33 INFO  : /chunks/0/4/4001_1_2570038: PUT from 127.0.0.1:58674

I tried both OneDrive and Local filesystem remote in rclone. Both of them fails.

It could be fixed by creating the file if not found before PUT operation maybe? But in most case it should be automatically created, but it is strange why this is not the case.

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version: juicefs version 1.0.0-beta1 (2022-01-13T15:45:45Z 28a42cf)
• Cloud provider or hardware configuration running JuiceFS:
• OS (e.g cat /etc/os-release): Windows 11
• Object storage (cloud provider and region, or self maintained): self-maintained
• Metadata engine info (version, cloud provider managed or self maintained): sqlite, just for test
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage): sqlite is local, object storage is "managed" by rclone (so it could go to OneDrive, iCloud, Backblaze, etc...)

[stevefan1999-personal]

I've tested an Rclone Linux WebDAV backend with JuiceFS Windows end. Somehow it works. So it's an Windows only issue.

[davies]

@stevefan1999-personal Without WebDAV, doest JuiceFS work in your Windows?

[stevefan1999-personal]

@stevefan1999-personal Without WebDAV, doest JuiceFS work in your Windows? of course

[zhijian-pro]

@stevefan1999-personal Sorry, I didn't reproduce this problem, my test result is that juicefs can use the webdav service exposed by rclone serve webdav as storage. I recommend compiling the latest executable tests using our main branch, and can first use the webdav service directly as storage for juicefs to rule out possible problems with rclone.

[zhijian-pro]

@stevefan1999-personal Which version of rclone are you using ? I was using rclone v1.57.0 when testing. It's work.

[stevefan1999-personal]

@stevefan1999-personal Which version of rclone are you using ? I was using rclone v1.57.0 when testing. It's work.

1.57.0

[davies]

Can't reproduce the issue, close it for now. Reopen it if needed.

[stevefan1999-personal]

@davies Oh because I switched to Linux and it works very well over there...