search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
10.16k stars 864 forks source link

Bug: file deletion triggers timeouts #2900

Closed DanySK closed 1 month ago

DanySK commented 1 month ago

Describe the bug

The system can download successfully from OneDrive. It can successfully upload on OneDrive. It gets stuck when a folder gets deleted, though.

Deleting item from Microsoft OneDrive: Apps
Internet connectivity to Microsoft OneDrive service has been interrupted .. re-trying in the background
A libcurl timeout has been triggered - data transfer too slow, no DNS resolution response, no server response
 - IPv6 DNS resolution issues may be causing timeouts. Consider setting 'ip_protocol_version' to IPv4 to potentially avoid this
 - HTTP/2 compatibility issues might also be interfering with your system. Use 'force_http_11' to switch to HTTP/1.1 to potentially avoid this
 - If these options do not resolve this timeout issue, please use --debug-https to diagnose this issue further.

this error occurs besides

onedrive --display-config                                                                                                                                                                                                                   Reading configuration file: <my user home>/.config/onedrive/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Application version                          = onedrive v2.5.2
Compiled with                                = DMD 2109
...
Config option 'ip_protocol_version'          = 1
...

Operating System Details

$ uname -a                                                                                                                                                                                                                                    Linux danidesktop-cesena 6.6.52-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Sep 18 21:00:34 UTC 2024 x86_64 GNU/Linux
$ cat /etc/lsb-release                                                                                                                                                                                                                        DISTRIB_ID="ManjaroLinux"
DISTRIB_RELEASE="24.1.0"
DISTRIB_CODENAME="Xahea"
DISTRIB_DESCRIPTION="Manjaro Linux"

Client Installation Method

From 3rd Party Source (PPA, OpenSuSE Build Service etc)

OneDrive Account Type

Business | Office365

What is your OneDrive Application Version

2.5.2

What is your OneDrive Application Configuration

Reading configuration file: /home/danysk/.config/onedrive/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Application version                          = onedrive v2.5.2
Compiled with                                = DMD 2109
User Application Config path                 = /home/danysk/.config/onedrive
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/danysk/.config/onedrive/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/danysk/.config/onedrive/sync_list
Applicable 'items.sqlite3' location          = /home/danysk/.config/onedrive/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = *Unsync*|*unsync*|.git|.gradle
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial|~*|.~*|*.tmp|*swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 1
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 8.10.0 (x86_64-pc-linux-gnu) libcurl/8.10.0 OpenSSL/3.3.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 nghttp3/1.5.0
Release-Date: 2024-09-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=16385176k,nr_inodes=4096294,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p3 on / type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10318)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-udev-load-credentials.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-tmpfiles-setup-dev-early.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-tmpfiles-setup-dev.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-vconsole-setup.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
/dev/nvme0n1p2 on /boot type ext4 (rw,relatime,data=ordered)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16398000k,nr_inodes=1048576,inode64)
/dev/sda1 on /datassd type ext4 (rw,noatime)
/dev/nvme1n1p1 on /datassd1tb type ext4 (rw,noatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/sdb1 on /datadisk type ext4 (rw,noatime,stripe=32744)
tmpfs on /run/credentials/systemd-tmpfiles-setup.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-sysctl.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3279596k,nr_inodes=819899,mode=700,uid=1000,gid=996,inode64)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=996)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=996)

What are all your local file system partition types

NAME        FSTYPE FSVER LABEL        UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                       
└─sda1      ext4   1.0                1897cfd6-1861-4dc9-a63b-6006a38803b6   48,8G    50% /datassd
sdb                                                                                       
└─sdb1      ext4   1.0                6489f818-8334-465d-ab1d-79d8ccbe42dc     44G    89% /datadisk
nvme0n1                                                                                   
├─nvme0n1p1 vfat   FAT32 UEFI_SYSTEM  4233-AFF8                             196,6M     0% /boot/efi
├─nvme0n1p2 ext4   1.0   AntergosBoot 62103300-87cd-4a3d-afea-73bf874601b5  115,3M    44% /boot
├─nvme0n1p3 ext4   1.0   AntergosRoot 1e8b40b4-95e4-431c-a623-9e7b8f3bce69  132,1G    65% /
└─nvme0n1p4 swap   1     AntergosSwap b2048b1e-c63d-4001-95c6-8a1fdeeb8dd5                [SWAP]
nvme1n1                                                                                   
└─nvme1n1p1 ext4   1.0   1TBSSD       036a99a9-131f-400f-8b77-9e6be0bdd66e  244,4G    68% /datassd1tb

How do you use 'onedrive'

Sync among Linux computers. No Windows, no Mac.

Steps to reproduce the behaviour

run a resync:

onedrive --resync --sync --verbose

then delete a folder from the synced onedrive folder

then sync again:

onedrive --sync --verbose

Complete Verbose Log Output

Deleted local items to delete on Microsoft OneDrive: 1
Deleting item from Microsoft OneDrive: Apps
Internet connectivity to Microsoft OneDrive service has been interrupted .. re-trying in the background
A libcurl timeout has been triggered - data transfer too slow, no DNS resolution response, no server response
 - IPv6 DNS resolution issues may be causing timeouts. Consider setting 'ip_protocol_version' to IPv4 to potentially avoid this
 - HTTP/2 compatibility issues might also be interfering with your system. Use 'force_http_11' to switch to HTTP/1.1 to potentially avoid this
 - If these options do not resolve this timeout issue, please use --debug-https to diagnose this issue further.
Retry attempt:           1 - Internal Thread ID: Jrk6iAuFL357WdT0
 This attempt timestamp: 2024-Oct-11 12:16:48
 Next retry in approx:   2 seconds
 Next retry approx:      2024-Oct-11 12:16:50
Retrying the respective Microsoft Graph API call for Internal Thread ID Jrk6iAuFL357WdT0 (Timestamp: 2024-Oct-11 12:16:50) ...
Internet connectivity to Microsoft OneDrive service has been interrupted .. re-trying in the background
A libcurl timeout has been triggered - data transfer too slow, no DNS resolution response, no server response
 - IPv6 DNS resolution issues may be causing timeouts. Consider setting 'ip_protocol_version' to IPv4 to potentially avoid this
 - HTTP/2 compatibility issues might also be interfering with your system. Use 'force_http_11' to switch to HTTP/1.1 to potentially avoid this
 - If these options do not resolve this timeout issue, please use --debug-https to diagnose this issue further.
Retry attempt:           2 - Internal Thread ID: Jrk6iAuFL357WdT0
 This attempt timestamp: 2024-Oct-11 12:17:57
 Next retry in approx:   4 seconds
 Next retry approx:      2024-Oct-11 12:18:01
Retrying the respective Microsoft Graph API call for Internal Thread ID Jrk6iAuFL357WdT0 (Timestamp: 2024-Oct-11 12:18:01) ...
Internet connectivity to Microsoft OneDrive service has been interrupted .. re-trying in the background
A libcurl timeout has been triggered - data transfer too slow, no DNS resolution response, no server response
 - IPv6 DNS resolution issues may be causing timeouts. Consider setting 'ip_protocol_version' to IPv4 to potentially avoid this
 - HTTP/2 compatibility issues might also be interfering with your system. Use 'force_http_11' to switch to HTTP/1.1 to potentially avoid this
 - If these options do not resolve this timeout issue, please use --debug-https to diagnose this issue further.
Retry attempt:           3 - Internal Thread ID: Jrk6iAuFL357WdT0
 This attempt timestamp: 2024-Oct-11 12:19:07
 Next retry in approx:   8 seconds
 Next retry approx:      2024-Oct-11 12:19:15

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 1 month ago

@DanySK This is not a bug:

image

Please be mindful of what constitutes a software bug and what does not.

You have a curl issue - and the application is telling you how to resolve this.

DanySK commented 1 month ago

Look, before opening this one I:

  1. verified that DNS resolution worked for both IPv6 and IPv4
  2. Applied the changes suggested in the guide, ip_protocol_version=1 and force_http_11=true

and I still experienced the issue. Unless I missed a clear compatibility table with curl, then I don't see how you can say this is not a bug. Call it a feature, if you like, but it is a problem preventing the application from working as intended -- regardless of whether it refers to a dependency.

Anyway, for those who run into the same issue, it looks like onedrive v2.5.2 requires curl 8.10.1. Today's Manjaro stable updates pulled this version in and the application returned to doing its job without needing to change any settings.

curl --version
curl 8.10.1 (x86_64-pc-linux-gnu) libcurl/8.10.1 OpenSSL/3.3.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 nghttp3/1.5.0
Release-Date: 2024-09-18
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zst
abraunegg commented 1 month ago

@DanySK

then I don't see how you can say this is not a bug.

A bug with 'curl' - yes .. a bug with this client - no ..

A 'bug' with this client would be:

but it is a problem preventing the application from working as intended

yes I agree with you that 'curl' prevented things from working correctly, and getting the latest version of 'curl' resolved your issue.

Look, before opening this one I:

Based on your application output you actually missed the downgrade to HTTP/1.1 .. In all of the past 5 years, these are the 3 things that have made the client 'work' again due to 'curl' bugs:

  1. Downgrade to IPv4 - this is due to the IPv6 DNS internal bug that 'curl' still has not fixed. This is the major contributor to your problem as 'curl' gets internally hung up because it forgets that IPv4 can be used
  2. Downgrade to HTTP/1.1 operations as the internal code for this in 'curl' has been way more vigorously tested by folk who use 'curl' than all the HTTP/2 code.
  3. Upgrading 'curl' (or in some cases downgrading as well) to fix things where the above two elements do not work.

This is why I call out these steps in Basic Troubleshooting Steps - so that folk that are running into odd problems, do some self diagnosis and problem solving first before hitting the panic button and raising a bug when they could have self helped.

Unless I missed a clear compatibility table with curl

No you did not miss something - however, I feel that this is potentially needed to list the known back versions of curl (generally all the Debian|Ubuntu versions) and will add 8.10.0 as well. Expect a documentation update that adds this to be added shortly.

DanySK commented 1 month ago

Thanks. I believe a compatibility table would be great, especially as this problem seems to come out often.

Would it make sense to switch the default options dynamically depending on the detected version of curl?

abraunegg commented 1 month ago

Would it make sense to switch the default options dynamically depending on the detected version of curl?

This would not have solved your issue however ..

Thanks. I believe a compatibility table would be great, especially as this problem seems to come out often.

Please review the following PR:

https://github.com/abraunegg/onedrive/blob/add-known-bad-curl-versions/docs/usage.md#compatibility-with-curl

abraunegg commented 3 weeks ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.