Bug: Upgraded from 2.4.25 to 2.5.2, not syncing I have applied the workarounds for curl

IanBlakeley commented 1 month ago

Describe the bug

After upgrading to latest I noticed files are not syncing and memory use was very high. I can see my curl version is one with problems so have applied the workarounds of forcing IPv4 and HTTP1.1 still after overnight I do not see that files are syncing. Memory use is rising again after several hours of use Capture

Operating System Details

Debian Bookworm

Linux Debian-VM1 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.2

What is your OneDrive Application Configuration

Reading configuration file: /home/ian/.config/onedrive/config
The option 'min_notify_changes' has been deprecated and will be ignored. Please read the updated documentation and update your client configuration to remove this option.
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Application version                          = onedrive v2.5.2
Compiled with                                = DMD 2104
User Application Config path                 = /home/ian/.config/onedrive
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/ian/.config/onedrive/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/ian/.config/onedrive/sync_list
Applicable 'items.sqlite3' location          = /home/ian/.config/onedrive/items.sqlite3
Config option 'drive_id'                     =
Config option 'sync_dir'                     = /home/ian/OneDrive.VM
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /home/ian/
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = Aragorn
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 1800
Config option 'monitor_log_frequency'        = 2
Config option 'monitor_fullscan_frequency'   = 24
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            =
Config option 'azure_tenant_id'              =
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2
Config option 'force_http_11'                = true
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 1
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false
Selective sync 'sync_list' configured        = false
Config option 'sync_business_shared_items'   = false
Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.14 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2023-02-20, security patched: 7.88.1-10+deb12u7
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

Filesystem                         Type      Size  Used Avail Use% Mounted on
udev                               devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs                              tmpfs     392M  2.8M  389M   1% /run
/dev/sda1                          ext4       28G  7.3G   19G  28% /
tmpfs                              tmpfs     2.0G  1.9M  2.0G   1% /dev/shm
tmpfs                              tmpfs     5.0M     0  5.0M   0% /run/lock
/dev/sda6                          ext4       97G   14G   79G  15% /home
tmpfs                              tmpfs     392M   60K  392M   1% /run/user/1001
tmpfs                              tmpfs     392M  140K  392M   1% /run/user/1030

What are all your local file system partition types

NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 ext4   1.0         7477e362-7125-4a1f-8b79-6613b5fdb36e   18.7G    26% /
├─sda2
├─sda5 swap   1           3c8ab922-8ad7-4de1-bd0a-0b8464645559                [SWAP]
└─sda6 ext4   1.0         1107f5d1-3e63-434b-bed5-5ae6ab276bfb     79G    13% /home
sr0
sr1

How do you use 'onedrive'

It is a folder on a Debian virtual machine running on my NAS so that it is on 24/7, the folder is shared so the desktop can write/read to it when it is used. The NAS backs up to a folder that is excluded from sync via its own client and their is a client on a Windows Laptop that also excludes the NAS folder

Steps to reproduce the behaviour

Seemed to be working prior to upgrading. I had previously upgraded to the pre-release of 2.5 but had issues so reverted. Cannot recall what the problem was.

Complete Verbose Log Output

Generated the debug log, it got pretty big and after a few hours the app binned itself. Log is zipped and a link emailed

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 1 month ago

@IanBlakeley Thanks for your debug file via email. That can now be removed online.

Essentially, there will be a few things happening - that will not be clear in the debug log you have provided. The issue will be clear when you perform HTTPS Debugging as well: https://github.com/abraunegg/onedrive/wiki/Generate-https-debug-log-for-support . What is happening is that a curl engine is being reused, but because it has not been reused in >120 seconds an upstream device from you (despite curl being told to keep sessions alive for reuse) - that upstream device closes the idle connection, and when the client goes to reuse the engine, it causes a SIGPIPE and exits. That is the most logical reason given the file quantities involved and the silent exit being seen. The HTTPS Debug log would confirm this.

Essentially:

You are still using a broken curl version
- As you are on Debian 12 I strongly advise you to upgrade curl|libcurl from the Debian 12 bookworm-backports repository
Manually build the client from 'master' which has this SIGPIPE fix
In your logs from API Response Bundle 2326 tells me you have a massive amount of data online. The way the client operates is 100% different to v2.4.x. You need to ensure you have enough memory / swap space on your platform for the client to operate. Please read: https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#memory-usage . This also adds weight to what I am thinking is your root cause (the SIGPIPE issue) as there is massive amounts of data to process - thus most certainly hitting this SIGPIPE issue.
Refer to https://github.com/abraunegg/onedrive/issues/2874 for the SIGPIPE issue

Actions for you:

Ensure your system is sized appropriately memory & swap space wise. You need 1GB of free physical memory per 100K items online. Please ensure your system has enough memory to operate. How v2.5.x operates is 100% different to v2.4.x.
Upgrade your curl version from Debian 12 bookworm-backports repository. This is strongly advised as an immediate action for you to take and will upgrade your curl version to 8.10.1.
Build your client manually from 'master' using onedrive v2.5.2-15-g77364c3 or greater to that you get the SIGPIPE fix to work around broken upstream network equipment from you that is terminating idle connections when it should not be.

IanBlakeley commented 1 month ago

Okay, thanks. I backported the curl version, downloaded the latest master but that's still reporting as 2.5.2 is that correct? There are a lot of files in that OneDrive but the majority are in an excluded folder the one the NAS uses for its backups there's only about 2500 files that should be synchronised. Anyhoo I've upped the memory to 6GB and will restart it and hope it doesn't run out of memory overnight

abraunegg commented 1 month ago

@IanBlakeley

but that's still reporting as 2.5.2 is that correct?

No that is not correct .. how are you cloning 'master' ? Your process is potentially incorrect.

This is a script I use:

#!/bin/bash

rm -rf ./onedrive/
git clone https://github.com/abraunegg/onedrive.git
cd onedrive

This will:

Clone 'master' into a directory called 'onedrive'

From there, you can:

Configure & build: ./configure --enable-debug --enable-notifications; make clean; make;
Test Version: ./onedrive --version

To install 'master' or any other version you must:

First uninstall any other version you have installed - it will be in a default path somewhere
Uninstall any 'onedrive' package you may have installed. Refer to your distribution instructions to uninstall

When ready to install:

Perform the following action: sudo make install from inside that local 'onedrive' directory

IanBlakeley commented 1 month ago

It did eventually sync overnight albeit it looks like I was skating close to running out of even 6GB of memory. I've just rebuilt as per you instructions above and it still reports 2.5.2 and looking back at the install

if [ -f .git/HEAD ] ; then \ git describe --tags > version ; \ else \ echo v2.5.2 > version ; \ fi

So I assume I've rebuilt the same version.

It looks like the large excluded folder and its subfolders get parsed despite not being synced.

abraunegg commented 1 month ago

You are probably doing something very basic wrong but as i dont know how you are doing things - the problem.could be anywhere.

I know the script I gave you clones the repo correctly .. are you cleaning up any old directory before cloning? What are all your commands that you are using?

abraunegg commented 1 month ago

It looks like the large excluded folder and its subfolders get parsed despite not being synced.

If you ever use --resync - then yes, otherwise no ... as you are doing a first run with v2.5.x then yes.

Please read all the documentation including the architecture documentation.

abraunegg commented 1 month ago

@IanBlakeley A for sure way also to know if you are running the right version:

When you use --display-config what does it show you ?

IanBlakeley commented 1 month ago

Thanks. in order of your queries.

Having got the code either by downloading the Zip or as per your earlier instruction cloning.

systemctl --user stop onedrive
source ~/dlang/dmd-2.104.0/activate
./configure --enable-notifications --enable-debug
make -j6 clean; make -j6;
sudo make install
systemctl --user daemon-reload
systemctl --user start onedrive
onedrive --version
systemctl --user status onedrive

Almost certainly then it is the first run. The excluded folder never comes near this virtual machine albeit it is on the same bare metal.

Display config shows 2.5.2 and the old curl version, there's is only one ccurl version on the device and only one onedrive

ian@Debian-VM1:~$ whereis onedrive
onedrive: /usr/local/bin/onedrive

ian@Debian-VM1:~$ whereis curl
curl: /usr/bin/curl /usr/share/man/man1/curl.1.gz

ian@Debian-VM1:~$ curl --version
curl 8.10.1 (x86_64-pc-linux-gnu) libcurl/8.10.1 GnuTLS/3.7.9 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 libssh2/1.10.0 nghttp2/1.52.0 ngtcp2/1.6.0 nghttp3/1.4.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2024-09-18, security patched: 8.10.1-1~bpo12+1
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

FWIW memory usage is now under 1% so it would seem that is the initial run that overloaded the device.

IanBlakeley commented 1 month ago

No idea why the curl text is struck through after posting

abraunegg commented 1 month ago

@IanBlakeley

Having got the code either by downloading the Zip

Do not do this .. you are only ever getting the last release .. not 'master' from Git.

This should be your process to download, configure and compile from source the client from master. The details below are from by Debian 12 system:

Step 1

Have a script, as 'root' user to ensure your system is correctly up-to-date. Review https://github.com/abraunegg/onedrive/blob/master/docs/ubuntu-package-install.md#step-2-ensure-your-system-is-up-to-date

Step 2

Reboot system post installing updates to ensure these are being used

Step 3

Install all of the required dependencies to compile the application under Debian 12 as per https://github.com/abraunegg/onedrive/blob/master/docs/install.md#dependencies-ubuntu-20x---ubuntu-24x--debian-10---debian-12---x86_64

sudo apt install build-essential -y
sudo apt install libcurl4-openssl-dev libsqlite3-dev pkg-config git curl -y
sudo apt install libnotify-dev -y 
curl -fsS https://dlang.org/install.sh | bash -s dmd

The DMD build step will install version 2.109.1 of the compiler as tested right now.

Step 4

Using the following script, download and build the client:

#!/bin/bash

rm -rf ./onedrive-master-github
git clone https://github.com/abraunegg/onedrive.git onedrive-master-github
cd onedrive-master-github

# Activate Compiler
source ~/dlang/dmd-2.109.1/activate

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;

# Deactivate Compiler
deactivate

# What was the version we just built
./onedrive --version

Step 5

Install the client from the 'onedrive-master-github' folder:

alex@debian-12-x64:~$ cd ./onedrive-master-github/
alex@debian-12-x64:~/onedrive-master-github$ sudo make install
[sudo] password for alex: 
/usr/bin/install -c -D onedrive /usr/local/bin/onedrive
/usr/bin/install -c -D -m 0644 onedrive.1 /usr/local/share/man/man1/onedrive.1
/usr/bin/install -c -D -m 0644 contrib/logrotate/onedrive.logrotate /usr/local/etc/logrotate.d/onedrive
mkdir -p /usr/local/share/doc/onedrive
/usr/bin/install -c -D -m 0644 readme.md config LICENSE changelog.md docs/advanced-usage.md docs/application-config-options.md docs/application-security.md docs/business-shared-items.md docs/client-architecture.md docs/contributing.md docs/docker.md docs/install.md docs/national-cloud-deployments.md docs/podman.md docs/privacy-policy.md docs/sharepoint-libraries.md docs/terms-of-service.md docs/ubuntu-package-install.md docs/usage.md docs/known-issues.md docs/webhooks.md /usr/local/share/doc/onedrive
/usr/bin/install -c -d -m 0755 /usr/lib/systemd/user /lib/systemd/system
/usr/bin/install -c -m 0644 contrib/systemd/onedrive@.service /lib/systemd/system
/usr/bin/install -c -m 0644 contrib/systemd/onedrive.service /usr/lib/systemd/user
alex@debian-12-x64:~/onedrive-master-github$ cd ..
alex@debian-12-x64:~$ onedrive --version
onedrive v2.5.2-16-g1efa4cf
alex@debian-12-x64:~$

abraunegg commented 1 month ago

I am closing this issue for the following reasons:

Curl bug
Issue is fixed in master

abraunegg commented 1 month ago

One last note - when the client is built correctly, installed correctly, and curl 8.10.1 is installed from Debian 12 Backports, when using onedrive --display-config the following is an example when using application defaults:

alex@debian-12-x64:~$ onedrive --display-config
Application version                          = onedrive v2.5.2-16-g1efa4cf
Compiled with                                = DMD 2109
Curl version                                 = libcurl/8.10.1 OpenSSL/3.0.14 zlib/1.3.1 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
User Application Config path                 = /home/alex/.config/onedrive
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/alex/.config/onedrive/config
Configuration file found in config location  = false - using application defaults
Applicable 'sync_list' location              = /home/alex/.config/onedrive/sync_list
Applicable 'items.sqlite3' location          = /home/alex/.config/onedrive/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2-16-g1efa4cf
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Config option 'max_curl_idle'                = 120
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false
alex@debian-12-x64:~$

abraunegg commented 1 month ago

FWIW memory usage is now under 1% so it would seem that is the initial run that overloaded the device.

As per:

https://github.com/abraunegg/onedrive/releases/tag/v2.5.0

As you jumped from v2.4.x -> v2.5.2 ... this note applies to you.

abraunegg commented 1 month ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

abraunegg / onedrive