search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
9.97k stars 859 forks source link

Bug: Non-systematic crashes on large repositories #2736

Closed robertschulze closed 3 months ago

robertschulze commented 4 months ago

Describe the bug

I use onedrive v2.5.0-rc2-25-g71bce90 and observe frequent crashes without any error message. The processing just stops, doesn't matter whether syncing or resyncing. Sometimes - without changing anything - it suddenly works.

Operating System Details

Linux atom 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.0-rc2-25-g71bce90

What is your OneDrive Application Configuration

onedrive version                             = onedrive v2.5.0-rc2-25-g71bce90
Config path                                  = /home/robert/.config/onedrive
Config file found in config path             = false
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc2-25-g71bce90
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.17
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Network

What are all your system 'mount points'

//hyperv/3 on /media/3 type cifs (rw,relatime,vers=3.1.1,cache=strict,username=Administrator,uid=1000,noforceuid,gid=100,noforcegid,addr=192.168.3.11,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

What are all your local file system partition types

NAME                  FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                 squashfs    4.0                                                         0   100% /snap/bare/5
loop1                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/264
loop2                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/278
loop3                 squashfs    4.0                                                         0   100% /snap/core/16574
loop4                 squashfs    4.0                                                         0   100% /snap/core/16928
loop5                 squashfs    4.0                                                         0   100% /snap/core18/2812
loop6                 squashfs    4.0                                                         0   100% /snap/core18/2823
loop7                 squashfs    4.0                                                         0   100% /snap/core20/2264
loop8                 squashfs    4.0                                                         0   100% /snap/core20/2318
loop9                 squashfs    4.0                                                         0   100% /snap/core22/1122
loop10                squashfs    4.0                                                         0   100% /snap/core22/1380
loop11                squashfs    4.0                                                         0   100% /snap/cups/1044
loop12                squashfs    4.0                                                         0   100% /snap/cups/1047
loop14                squashfs    4.0                                                         0   100% /snap/firefox/4259
loop15                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/104
loop16                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/111
loop17                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/194
loop18                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/198
loop19                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/90
loop20                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/93
loop21                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/140
loop22                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/143
loop23                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/172
loop24                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/176
loop25                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/184
loop26                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/186
loop27                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1534
loop28                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1535
loop29                squashfs    4.0                                                         0   100% /snap/snap-store/1113
loop30                squashfs    4.0                                                         0   100% /snap/snap-store/959
loop31                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/157
loop32                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/83
loop33                                                                                        0   100% /snap/firefox/4336
sda                                                                                                    
├─sda1                vfat        FAT32          62EB-8E64                               503,3M     1% /boot/efi
├─sda2                                                                                                 
└─sda5                LVM2_member LVM2 001       X34xd2-HOp4-sAHk-pudA-CzVG-rENF-FOtrO2                
  ├─ubuntu--vg-root   ext4        1.0            43f5fc57-8370-4315-b5ba-95912f21e113     20,8G    74% /var/snap/firefox/common/host-hunspell
  │                                                                                                    /media/robert_home
  │                                                                                                    /
  └─ubuntu--vg-swap_1 swap        1              ad32bbec-507e-44b0-bb5f-1df3ee647a2a                  [SWAP]

How do you use 'onedrive'

./onedrive --confdir='/home/robert/.config/onedrive/accounts/robert@guitaronline.de' --sync --verbose --verbose --resync --resync-auth > debug_output.log 2>&1 or ./onedrive --confdir='/home/robert/.config/onedrive/accounts/robert@guitaronline.de' --sync --verbose --verbose > debug_output.log 2>&1

Steps to reproduce the behaviour

run as above

Complete Verbose Log Output

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 258 of 258 as part of JSON Item Batch 6 of 6
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"aYzpGMzAyOEJENzU4NTUyRkFGITQyMDEuMjU3","eTag":"aRjMwMjhCRDc1ODU1MkZBRiE0MjAxLjM4","file":{"hashes":{"quickXorHash":"g87ffeQiyBc7oVri8W2sVUFb2Hk=","sha1Hash":"A1E6035436A12CC51952EDED1405DD28F619DBEB","sha256Hash":"77202CA8100E423DBDD3F63C08B797E0F15DB369618482032B3E829144D7FD6D"},"mimeType":"application\/vnd.ms-excel"},"fileSystemInfo":{"createdDateTime":"2022-11-14T00:02:07.866Z","lastModifiedDateTime":"2008-10-22T20:31:24Z"},"id":"F3028BD758552FAF!4201","name":"Stundenplan.xls","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!3668","name":"Studium","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!4690:\/Studium"},"size":17408}
DEBUG: Attempting to calculate local filesystem path for f3028bd758552faf and F3028BD758552FAF!3668
DEBUG: JSON Item calculated full path is: OneDrive - Constanze/Studium/Stundenplan.xls
DEBUG: The item we are syncing is a file
DEBUG: skip_file item to check: /OneDrive - Constanze/Studium/Stundenplan.xls
DEBUG: skip_file evaluation for: /OneDrive - Constanze/Studium/Stundenplan.xls
DEBUG: Result: false
DEBUG: sync_list item to check: OneDrive - Constanze/Studium/Stundenplan.xls
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Constanze/Studium/Stundenplan.xls
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a file
DEBUG: OneDrive change is potentially a new local item
DEBUG: Batched JSON item processing time: 655 μs and 3 hnsecs
<here it just stops>

or:

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 262 of 262 as part of JSON Item Batch 6 of 6
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpGMzAyOEJENzU4NTUyRkFGITQyNTAuNjM4NDA4NjY1MDU5OTMwMDAw","eTag":"aRjMwMjhCRDc1ODU1MkZBRiE0MjUwLjI","fileSystemInfo":{"createdDateTime":"2023-03-06T16:00:36.51Z","lastModifiedDateTime":"2023-11-06T20:40:17.653Z"},"id":"F3028BD758552FAF!4250","name":"Notizbuch von Constanze","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!4690","name":"OneDrive - Constanze","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!4690:"},"size":1344676}
DEBUG: Attempting to calculate local filesystem path for f3028bd758552faf and F3028BD758552FAF!4690
DEBUG: JSON Item calculated full path is: OneDrive - Constanze/Notizbuch von Constanze
DEBUG: The Microsoft OneNote Notebook 'OneDrive - Constanze/Notizbuch von Constanze' is not supported by this client
DEBUG: Flagging as unwanted: item type is not supported
DEBUG: Skipping OneDrive change as this is determined to be unwanted
<here it just stops>

Screenshots

No response

Other Log Information or Details

I will send the full log files via mail.

Additional context

No response

abraunegg commented 4 months ago

@robertschulze Your RC2 version is onedrive v2.5.0-rc2-25-g71bce90

The latest correct RC2 version is onedrive v2.5.0-rc2-24-gfb77411 .. which branch are you actually running?

The correct branch has this commit: https://github.com/abraunegg/onedrive/pull/2709/commits/ac334e8860aefc0a424d596e2a684f81efe5ffe3 which changes the Curl handling for signalling which can be the cause of this.

As you are using RC2, I would also suggest you test PR #2728 - it has a significant amount of fixes in in and will probably become the base for RC3 (which then I will look at your other issue). When using #2728 your version will be onedrive v2.5.0-rc2-37-g3f7fb5a at the moment.

The last thing here is memory on the system .. check that the application is not being killed by an OOM process.

I am constantly testing all of these versions with 125K files online. What I have also found is that memory usage, when using Ubuntu versions & the Ubuntu 'ldc' provided version, it is way higher than using the 'dmd' or 'ldc' package install using the install script - so something else to think about.

Actions for you:

  1. Check what branch you are building from
  2. Test PR #2728
robertschulze commented 3 months ago

@abraunegg I used PR #2728 (onedrive v2.5.0-rc2-37-g3f7fb5a) but unfortunately the problem still persists. I will send four log files from four subsequent runs that just stop at entirely different steps. For the last two it is after the processing of the last JSON item of the last JSON Item Batch.

The thing is that there is no error message or anything, they just stop. Is there anything I can do to better investigate it?

Memory appears to be no issue, usage according to top is 0.3% during the whole time.

abraunegg commented 3 months ago

@robertschulze

The thing is that there is no error message or anything, they just stop. Is there anything I can do to better investigate it?

Do not use Ubuntu? Is that an option?

A couple of things for you to test:

  1. Force the client to use IPv4 only and to use HTTP/1.1 only 
    ip_protocol_version = "1"
force_http_11 = "true"
  2. Upgrade your curl version to something way more modern than curl 7.81.0
  3. Perform HTTP Debug Logging Output - add: --verbose --verbose --debug-https to whatever command is executing.
  4. Upgrade your Ubuntu version to 24.04, then upgrade your curl version
  5. Change your OS to Debian 12 at a starting point
  6. Move to something that is not based on Ubuntu

Something upstream is potentially doing something ... but I am not ruling out some of the other Curl changes at this stage - but the HTTPS Debug Logs will show this. Please read: https://github.com/abraunegg/onedrive/wiki/Generate-https-debug-log-for-support

No other log other than the HTTPS Debug Log is going to assist here.

I have been doing >72Hr long runs with earlier code from #2728 on the following platforms using the default configuration:

Everything is rock solid .. nothing like this is being seen. So right now I have to point the finger at your environment / network path between you and Microsoft OneDrive. The HTTPS Debug Log will give more clues.

abraunegg commented 3 months ago

@robertschulze Any update here ?

robertschulze commented 3 months ago

@abraunegg Yes, I tested the ideas:

  1. Force to IPv4 and HTTP/1.1: same result, the only thing that changed was that the log file stops one (simple separator) line before the last iteration: image
  2. With HTTP Debug Output Logging the separator line is back, and some additional lines: image
  3. With curl having been upgraded to 
    Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

    the code continues past the old breaking point: image i.e. the line DEBUG: Number of JSON items processed was: 2759 appears. So the breaking point must have been between the two log entries: 

    DEBUG: Number of JSON items to process is: 2759
DEBUG: Number of JSON items processed was: 2759
  4. /5./6. Unfortunately, I cannot (yet) upgrade to Ubuntu 24 because I am on the LTS branch which I would greatly prefer to remain on. But yes, with with all the experience and issues here I guess I should consider switching my whole NAS setup to Fedora or such in the long run...
abraunegg commented 3 months ago

@robertschulze

Force to IPv4 and HTTP/1.1: same result, the only thing that changed was that the log file stops one (simple separator) line before the last iteration

That is an odd one as this has nothing to do with that option being set. It could be console logging output just missing it due to volume of logging done .. as any separation line is a standard line output .. thats just an odd one an I am scratching my head on that.

With HTTP Debug Output Logging the separator line is back, and some additional lines

OK .. so the reason the Curl engine was stopping is that something upstream from you closed the active connection. That is the detail needed when using --debug-https that helps understand why something is being killed.

As you are on Ubuntu the finger pointing starts at Ubuntu and its horrid ufw and its settings there. From there you need to look at your router and how you connect to the Internet in general, then speak to your ISP as to what is closing connections. I would be starting with ufw and get rid of that awful inclusion - or at least configure it so that it is not killing connections.

With curl having been upgraded to

What Curl version did you upgrade to ? By the date, it looks like you may be using 8.8.0 - this is a good thing, so at least all known issues post curl 7.81.0 will have been resolved

Post upgrading Curl - are things working or not working for you now - this is unclear from what you have posted

Older Curl versions might still be using internal code that does not adhere correctly to CURLOPT_NOSIGNAL - hopefully all Curl libraries and tools were correctly updated on your system when you did the upgrade.

Unfortunately, I cannot (yet) upgrade to Ubuntu 24 because I am on the LTS branch which I would greatly prefer to remain on.

Ubuntu 24.x is also an LTS Branch , but if you were going to rebuild - I would suggest in order:

robertschulze commented 3 months ago

@abraunegg I ran a few more tests and with the updated curl (8.8.0 indeed) it appears to work reliably now, no more sudden code stops. Thanks!

abraunegg commented 3 months ago

@abraunegg I ran a few more tests and with the updated curl (8.8.0 indeed) it appears to work reliably now, no more sudden code stops. Thanks!

Great .. so yet another Ubuntu special ... keeping users on old versions in the guise of stability actually causing stability issues .. who would have thought that ......

abraunegg commented 3 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.