Open christianbur opened 10 months ago
@christianbur hello, which webdav client are you using?
Webdav-Client: https://www.photosync-app.com/
Hello, we will conduct a test with this client next week. If there are any further info, we will notify you.
same error https://forum.seafile.com/t/webdav-upload-big-files/4546
I tested webdav again today with http:// (without encryption, without loadbalancer), but again the same problem with the MOV files.
I have hit this as well. Log file below. It has nothing to do with the size of the file. The title of this issue should be changed.
In my case it's just a small and innocent PDF attached here: Note1.pdf
WebDAV client is an Onyx Boox Tab X (e-ink tablet) which WORKS FINE when using the native Apache WebDAV, instead of the seafile webdav.
It seems the problem is that the wsgidav server expects the data in chunks, and the client setting/not setting a HTTP header that the wsgidav server doesn't/does expect, and the wsgidav code ends up trying to cast the first few bytes of the file to int (note the b'%PDF-1.7\r\n'
which are the first bytes in the PDF file), see code at https://github.com/haiwen/seafdav/blob/4f52d8d0415d5c364c23e981c1c52547c36fe121/wsgidav/request_server.py#L686
... not sure if the fault is with wsgidav or whatever Seafile code that intermediates this. I'm not familiar with wsgidav to debug further. At least the HTTP headers passed to it should be investigated, since the same device does work fine with the native Apache webdav server.
My seafile setup is a Docker setup v10.0.1 behind an Apache reverse proxy (but that doesn't make a difference). I'm including the apache conf and the docker compose conf below as well, for reference. Seafile works fine otherwise. It's just this uploading from the Boox device that fails to seafdav (but which works fine to Apache's native webdav).
seafdav.conf
[WEBDAV]
enabled = true
port = 8080
share_name = /dav
seafdav.log:
02:30:40.172 - INFO : 172.18.0.1 - (anonymous) - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/" length=93, depth=0, elap=0.000sec -> 401 Not Authorized
02:30:40.265 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/" length=93, depth=0, elap=0.049sec -> 207 Multi-Status
02:30:40.392 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/" length=96, depth=0, elap=0.041sec -> 207 Multi-Status
02:30:40.491 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/TabX/" length=93, depth=0, elap=0.051sec -> 207 Multi-Status
02:30:40.629 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/TabX/" length=96, depth=0, elap=0.053sec -> 207 Multi-Status
02:30:40.764 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/TabX/Notepads/" length=93, depth=0, elap=0.048sec -> 207 Multi-Status
02:30:40.898 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:40] "PROPFIND /lib/onyx/TabX/Notepads/" length=96, depth=0, elap=0.049sec -> 207 Multi-Status
02:30:41.100 - ERROR : PUT: byte copy failed
Traceback (most recent call last):
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_server.py", line 837, in do_PUT
fileobj.writelines(data_stream)
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_server.py", line 686, in _stream_data_chunked
length = int(buf, 16)
ValueError: invalid literal for int() with base 16: b'%PDF-1.7\r\n'
02:30:41.100 - ERROR : Caught HTTPRequestException(HTTP_INTERNAL_ERROR)
Traceback (most recent call last):
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_server.py", line 837, in do_PUT
fileobj.writelines(data_stream)
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_server.py", line 686, in _stream_data_chunked
length = int(buf, 16)
ValueError: invalid literal for int() with base 16: b'%PDF-1.7\r\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/error_printer.py", line 50, in __call__
for v in app_iter:
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_resolver.py", line 224, in __call__
for v in app_iter:
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_server.py", line 127, in __call__
app_iter = provider.custom_request_handler(environ, start_response, method)
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/dav_provider.py", line 1594, in custom_request_handler
return default_handler(environ, start_response)
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/request_server.py", line 847, in do_PUT
util.fail(e)
File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/util.py", line 949, in fail
raise e
wsgidav.dav_error.DAVError: 500
02:30:41.100 - ERROR : e.src_exception:
invalid literal for int() with base 16: b'%PDF-1.7\r\n'
02:30:41.100 - INFO : 172.18.0.1 - username@example.com - [2024-01-23 02:30:41] "PUT /lib/onyx/TabX/Notepads/Note1.pdf" elap=0.069sec -> 500 Internal Server Error
apache conf:
<VirtualHost *:80>
ServerName sf.example.com
Redirect permanent / https://sf.example.com/
</VirtualHost>
<VirtualHost *:443>
ServerName sf.example.com
ServerAdmin noreply@example.com
# Possible values: trace8,..., trace1, debug, info, notice, warn, error, crit, alert, emerg.
LogLevel notice
#LogLevel notice rewrite:trace3
ErrorLog ${APACHE_LOG_DIR}/sf-error.log
CustomLog ${APACHE_LOG_DIR}/sf-access.log combined
#Protocols h2 http/1.1
ProxyVia On
ProxyRequests Off
AllowEncodedSlashes NoDecode
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
ProxyPreserveHost On
ProxyPass / http://localhost:20080/
ProxyPassReverse / http://localhost:20080/
<Location /dav>
RequestHeader edit Destination ^https: http:
ProxyPass http://localhost:28080/dav
ProxyPassReverse http://localhost:28080/dav
LogLevel trace3
</Location>
SSLEngine on
SSLCertificateFile /etc/ssl/acme.sh/example.com.fullchain.pem
SSLCertificateKeyFile /etc/ssl/acme.sh/example.com.key.pem
</VirtualHost>
docker-compose.yml
services:
db:
image: mariadb:10.11
container_name: seafile-mysql
environment:
- MYSQL_ROOT_PASSWORD=blabla # Requested, set the root's password of MySQL service.
- MYSQL_LOG_CONSOLE=true
volumes:
- /var/seafile/mysql/db:/var/lib/mysql # Requested, specifies the path to MySQL data persistent store.
networks:
- seafile-net
memcached:
image: memcached:1.6.18
container_name: seafile-memcached
entrypoint: memcached -m 256
networks:
- seafile-net
seafile:
restart: always
image: seafileltd/seafile-mc:latest
container_name: seafile
ports:
- 20080:80
- 28080:8080 # seafile WebDAV
#- 20443:443 # If https is enabled, cancel the comment.
extra_hosts:
- host.docker.internal:host-gateway # allow the docker to reach host servies by using the dns name "host.docker.internal" instead of extracting the host's IP address 172
volumes:
- /var/seafile/data:/shared # Requested, specifies the path to Seafile data persistent store.
environment:
- DB_HOST=db
- DB_ROOT_PASSWD=blabla # Requested, the value should be root's password of MySQL service.
- TIME_ZONE=Etc/UTC # Optional, default is UTC. Should be uncomment and set to your local time zone.
- SEAFILE_ADMIN_EMAIL=seafile.admin@example.com # Specifies Seafile admin user, default is 'me@example.com'.
- SEAFILE_ADMIN_PASSWORD=blabla # Specifies Seafile admin password, default is 'asecret'.
- SEAFILE_SERVER_LETSENCRYPT=false # Whether to use https or not.
- SEAFILE_SERVER_HOSTNAME=sf.example.com # Specifies your host name if https is enabled.
- FORCE_HTTPS_IN_CONF=true # if you want a reverse proxy with its own ssl cert then enable this, and SEAFILE_SERVER_LETSENCRYPT as well as port 443:443
depends_on:
- db
- memcached
networks:
- seafile-net
networks:
seafile-net:
Add to my message above, I did some debugging ... as I suspected, the problem is with files served as chunked.
wsgidav is trying to extract the chunk length from the first bytes of the chunk (length = int(buf, 16)
at line 686 in request_server.py), which according to https://en.wikipedia.org/wiki/Chunked_transfer_encoding is correct, but this client is not prepending the chunk size. In addition, apparently there is no chunked transfer in HTTP/2.0, but this client is using PUT HTTP/2.0 with Transfer-Encoding: chunked.
EDIT: actually, the webdav clients are fine, the underlying SGI server already de-chunked the data, so WsgiDAV should't try to de-chunk it again. This is a bug in WsgiDAV.
HOWEVER, Apache's native webdav doesn't have an issue with it, it works fine. Neither does Nextcloud's webdav. It may be that when there is no chunk size prepended to the chunk data then the server simply treats it as un-chunked.
The iPhone client from @christianbur also (almost surely) behaves the same, i.e. doesn't prepdend the chunk size to the chunk data like wsgidav expects. This can be easily patched.
Request headers from the Boox device (NOTE THE Transfer-Encoding: chunked
):
PUT /dav/lib/onyx/TabX/Notepads/Note1.pdf HTTP/2.0
Mac: 00%3a00%3a00%3a00%3a00%3a00
Version: 30
Deviceuniqueid: 3000000000000
Buildid: 2023-11-22_10-35_3.5_946657f755
Language: eu
Device-Lang:en_US
Authorization: Basic YXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
Content-Type: application/pdf
Accept-Encoding: gzip
User-Agent: okhttp/4.10.0
Host: sf.example.com
Transfer-Encoding: chunked
Request headers from curl -v --basic --user "username:password" -T ~/Downloads/Note1.pdf "https://sf.example.com/dav/lib/Note1.pdf"
:
PUT /dav/lib/Note1.pdf HTTP/2.0
Authorization: Basic YXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
User-Agent: curl/8.5.0
Accept: */*
Content-Length: 84003
Host: sf.example.com
WAIT ... you are using a version of WsgiDAV that is too old!
You're using WsgiDAV 4.1.0 but WsgiDAV is currently at 4.3.0 and there is an older commit that addresses this very chunked data issue: https://github.com/mar10/wsgidav/commit/dd3e53a5c57b03c19d1d240fbe949b8f75dd6850 (see original PR here https://github.com/mar10/wsgidav/pull/291)
Can you please push an updated Docker image with WsgiDAV 4.3.0 ?
... there goes 3h of my life
@imwhatiam see above - you need to update WsgiDAV to 4.3.0
We will check the issue.
@freeplant and @imwhatiam I created a PR (commit changes here) that you can integrate into the current version of WsgiDAV without needing to upgrade WsgiDAV to 4.3.0. You can upgrade WsgiDAV to 4.3.0 later when you have time to check if it breaks integration with Seafile, they may have changed command line arguments in 4.3.0.
@christianbur and others who would like to fix it before Seafile devs release a new version: You can replace seahub/thirdpart/wsgidav/request_server.py
with the patched from my repo here (applies to all current v10 and v11 versions of Seafile): https://github.com/aleqx/seafdav/blob/master/wsgidav/request_server.py
If you're using Docker then you can just replace that same file inside the docker image (depending on how profficient with Docker you are). I also uploaded patched docker images to my docker hub for the current v10 and v11 here (you need to change the image name in your docker-compose.yml): https://hub.docker.com/repository/docker/aleqxdocker/seafile-mc/tags
@imwhatiam see above - you need to update WsgiDAV to 4.3.0
Hello, I just upgraded seafdav to use wsgidav 4.3, and deployed the upgraded seafdav code to our demo site. Could you please connect to demo.seafile.com/seafdav/ using your WebDAV client and test it?
Username: admin@seafile.local
Password: epaeMah5
Doesn't work. I'm not getting a specific error (it fails differently to how it was failing under 4.1.0). My device needs to create folders when uploading (it attempts to create the folder /onyx
in the root of the DAV account (under /seafdav
) so you may want to check your dav logs to see if there were any such attempts. You may need to give more permissions.
I could open that dav folder in a browser, and login, and check that there was no successful upload from my device.
I created a PR (commit changes here)
Does the pull request successful solve your issue after you manually modified the code in your Seafile?
Yes, my PR works fine (wouldn't have created a PR without testing it first). I'm using it in a patched docker 10.0.1
My device needs to create folders when uploading (it attempts to create the folder
/onyx
in the root of the DAV account (under/seafdav
) so you may want to check your dav logs to see if there were any such attempts.
Hello, this issue is likely arosed after Seafdav was rebased on Wsgidav 4.3. We will address and rectify this problem.
I use https://www.photosync-app.com to backup my photos and videos from my iPhone. I initially used Nextcloud with Webdav (no problems) and have now switched to Seafile. I can upload all photos using Webdav without any problems, but no live photos and videos, both use MOV files. Why is it not possible to upload MOV files? I use haproxy as a load balancer for https. I use Seafile with Docker (image: seafileltd/seafile-mc:latest)
./seafile/conf/seafdav.conf INFO: Tested with "fastcgi = true" and without
./seafile/logs/seafdav.log