search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
9.97k stars 859 forks source link

Bug: Real-time synchronization is not occurring when using --monitor and sync_list #2832

Closed kayzzen01 closed 1 week ago

kayzzen01 commented 2 weeks ago

Describe the bug

Real-time synchronization is not occurring, changes are only processed based on the 'monitor_interval' time.

Operating System Details

Linux arch 6.6.52-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 18 Sep 2024 19:02:04 +0000 x86_64 GNU/Linux

LSB Version:    n/a
Distributor ID: Arch
Description:    Arch Linux
Release:        rolling
Codename:       n/a

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

2.5.0

What is your OneDrive Application Configuration

D-Bus message bus daemon is available; GUI notifications are now enabled
Reading configuration file: /home/kayzzen01/.config/onedrive/config
Configuration file successfully loaded
Application version                          = onedrive v2.5.0
Compiled with                                = LDC 2109
User Application Config path                 = /home/kayzzen01/.config/onedrive
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/kayzzen01/.config/onedrive/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/kayzzen01/.config/onedrive/sync_list
Applicable 'items.sqlite3' location          = /home/kayzzen01/.config/onedrive/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial|~*|.~*|*.tmp|*.kate-swp|*.part
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 104857600
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = true
sync_list config option 'sync_root_files'    = false
sync_list contents:
# sync_list supports comments
#
# The ordering of entries is highly recommended - exclusions before inclusions
#
# Exclude temp folders under Documents

# Include OneDrive Root Folders
/Config
/Documents
/GamesAll
/Music
/Pictures
/Programs
/Videos
/Workspace

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 8.10.1 (x86_64-pc-linux-gnu) libcurl/8.10.1 OpenSSL/3.3.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 nghttp3/1.5.0
Release-Date: 2024-09-18
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=16362236k,nr_inodes=4090559,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme1n1p2 on / type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10322)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-udev-load-credentials.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-sysctl.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-tmpfiles-setup-dev-early.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-tmpfiles-setup-dev.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-vconsole-setup.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16381344k,nr_inodes=1048576,inode64)
/dev/nvme1n1p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/nvme1n1p3 on /home type ext4 (rw,relatime)
tmpfs on /run/credentials/systemd-tmpfiles-setup.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3276268k,nr_inodes=819067,mode=700,uid=1000,gid=1000,inode64)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

What are all your local file system partition types

NAME        FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                
nvme1n1                                                                            
├─nvme1n1p1 vfat   FAT32 efi   0B35-03F7                               872M    15% /boot
├─nvme1n1p2 ext4   1.0   root  4c864fdd-c3a3-4b3f-859e-5ac05e620795  102,2G    12% /
├─nvme1n1p3 ext4   1.0   home  4593b448-6a87-4ff5-af46-39b79c1cf3f4  530,1G    33% /home
└─nvme1n1p4 swap   1     swap  625bdf57-e1de-4290-a1b2-bc66b2fb50a5                [SWAP]
nvme0n1                                                                            
├─nvme0n1p1                                                                        
└─nvme0n1p2 ntfs               523E52BE3E529B35

How do you use 'onedrive'

I only use it on my personal desktop. I'm not using it on Windows and I have a cell phone that syncs photos to OneDrive.

Steps to reproduce the behaviour

When updating/modifying or renaming or deleting a file. These modifications will only be reflected in OneDrive when the 'monitor_interval' is performed.

Complete Verbose Log Output

[2024-Sep-19 20:02:44.9791963] Using the following path to store the runtime application log: /var/log/onedrive/
[2024-Sep-19 20:02:44.9796856] Using IPv4 and IPv6 (if configured) for all network operations
[2024-Sep-19 20:02:44.9802014] Attempting to contact Microsoft OneDrive Login Service
[2024-Sep-19 20:02:45.1875510] Successfully reached Microsoft OneDrive Login Service
[2024-Sep-19 20:02:45.1883463] Checking Application Version ...
[2024-Sep-19 20:02:45.4817874] Attempting to initialise the OneDrive API ...
[2024-Sep-19 20:02:45.4818520] Configuring Global Azure AD Endpoints
[2024-Sep-19 20:02:45.4818900] The OneDrive API was initialised successfully
[2024-Sep-19 20:02:45.4821311] Opening the item database ...
[2024-Sep-19 20:02:45.4848861] User Configured Rate Limit: 104857600
[2024-Sep-19 20:02:52.4713271] Application Version:  onedrive v2.5.0
[2024-Sep-19 20:02:52.4713447] Account Type:         personal
[2024-Sep-19 20:02:52.4713520] Default Drive ID:     43AFED0000C6C4F0
[2024-Sep-19 20:02:52.4713563] Default Root ID:      43AFED0000C6C4F0!sea8cc6beffdb43d7976fbc7da445c639
[2024-Sep-19 20:02:52.4713633] Remaining Free Space: 926.60 GB (994929174118 bytes)
[2024-Sep-19 20:02:52.4713666] Sync Engine Initialised with new Onedrive API instance
[2024-Sep-19 20:02:52.4716702] All application operations will be performed in the configured local 'sync_dir' directory: /home/kayzzen01/OneDrive
[2024-Sep-19 20:02:52.4718151] OneDrive synchronisation interval (seconds): 300
[2024-Sep-19 20:02:52.4718257] Maximum allowed open files:                  9223372036854775807
[2024-Sep-19 20:02:52.4718299] Maximum allowed inotify user watches:        524288
[2024-Sep-19 20:02:52.4718337] Initialising filesystem inotify monitoring ...
[2024-Sep-19 20:02:52.4847718] Monitoring directory: .
[2024-Sep-19 20:02:52.4860285] Performing initial synchronisation to ensure consistent local state ...
[2024-Sep-19 20:02:52.4860650] Attempting to contact Microsoft OneDrive Login Service
[2024-Sep-19 20:02:52.6704129] Successfully reached Microsoft OneDrive Login Service
[2024-Sep-19 20:02:52.6715288] Starting a sync with Microsoft OneDrive
[2024-Sep-19 20:02:52.9930310] Fetching /delta response from the OneDrive API for Drive ID: 43AFED0000C6C4F0
[2024-Sep-19 20:02:53.2344276] Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
[2024-Sep-19 20:02:53.2351819] Finished processing /delta JSON response from the OneDrive API
[2024-Sep-19 20:02:53.2354825] No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
[2024-Sep-19 20:02:53.2418541] Performing a database consistency and integrity check on locally stored data
[2024-Sep-19 20:02:53.2441782] Processing DB entries for this Drive ID: 43AFED0000C6C4F0
[2024-Sep-19 20:02:53.5242019] Processing: ~/OneDrive
[2024-Sep-19 20:02:53.5242183] The directory has not changed
[2024-Sep-19 20:08:00.2704567] Changed local items to upload to Microsoft OneDrive: 1
[2024-Sep-19 20:08:01.2982681] Uploading modified file: Config/teste-file.txt ... done
[2024-Sep-19 20:08:01.6457206] Scanning the local file system '~/OneDrive' for new data to upload
[2024-Sep-19 20:08:03.4256177] Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
[2024-Sep-19 20:08:03.4257839] Fetching /delta response from the OneDrive API for Drive ID: 43AFED0000C6C4F0
[2024-Sep-19 20:08:03.6947448] Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3
[2024-Sep-19 20:08:03.6951574] Skipping path - excluded by sync_list config: ./Config
[2024-Sep-19 20:08:03.6954398] Including file - included by sync_list config: Config/teste-file.txt
[2024-Sep-19 20:08:03.6964252] Finished processing /delta JSON response from the OneDrive API
[2024-Sep-19 20:08:03.6968947] Processing 1 applicable changes and items received from Microsoft OneDrive
[2024-Sep-19 20:08:03.6968991] Processing OneDrive JSON item batch [1/1] to ensure consistent local state
[2024-Sep-19 20:08:03.7025552] Sync with Microsoft OneDrive is complete
[2024-Sep-19 20:08:54.0919731] Received termination signal, attempting to cleanly shutdown application
[2024-Sep-19 20:08:54.0920362] Monitored directory removed: ./
[2024-Sep-19 20:08:54.0921215] Waiting for any existing upload|download process to complete
[2024-Sep-19 20:08:54.0925628] Attempting to perform a database vacuum to optimise database
[2024-Sep-19 20:08:54.1624116] Database vacuum is complete

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 2 weeks ago

@kayzzen01

Please can you compile the following PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2838/head:pr2838
git checkout pr2838

To build the PR, please follow the steps to prepare your system to build the client as detailed here: https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements

When building the client, please use the following:

./configure --enable-debug; make clean; make;

When running the PR, your version should be: onedrive v2.5.0-6-g1a5e0b3 or greater.

I have tested this PR with the following, using your 'sync_list' configuration:

./onedrive -m -v
......
[M] New local file added: Documents/new_local_file2.txt
New items to upload to Microsoft OneDrive: 1
Total New Data to Upload:        21 Bytes
Uploading new file: Documents/new_local_file2.txt ... done
[M] Total number of local file(s) added or changed: 1
........

The result is that the new local file, is correctly picked up, and uploaded within a couple of seconds to Microsoft OneDrive as per expectation.

abraunegg commented 2 weeks ago

@kayzzen01

Please can you respond

kayzzen01 commented 2 weeks ago

@abraunegg,

Sorry for the delay. Today has been a busy day. In an hour I will carry out the procedures you requested.

@kayzzen01

Please can you respond

kayzzen01 commented 2 weeks ago

Hi @abraunegg,

The syncs are being performed within about 1 second of making any modification. As per your evidence above.

.....
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
[M] Local file changed: Config/text-file-new01.txt
Uploading modified file: Config/text-file-new01.txt ... done
[M] Total number of local file(s) added or changed: 1
.....

Working as expected! :clap:

abraunegg commented 2 weeks ago

Can you also confirm, based on prior experience with v2.4.x - this is acting as it did before v2.5.0

Im just trying to confirm i've also not now inadvertantly broken something else with 'sync_list' operations with this fix.

Any feedback on that is greatly appreciated.

kayzzen01 commented 2 weeks ago

Yes, version 2.5.0 is working similarly to version 2.4.x.

I don't know if it's relevant to the case. I say similar because I'm finding version 2.5.0 faster than version 2.4.x in synchronizing events.

Sep 22 09:45:53 arch onedrive[814]: Attempting to contact Microsoft OneDrive Login Service
Sep 22 09:45:54 arch onedrive[814]: Successfully reached Microsoft OneDrive Login Service
Sep 22 09:45:54 arch onedrive[814]: Starting a sync with Microsoft OneDrive
Sep 22 09:45:54 arch onedrive[814]: Syncing changes from Microsoft OneDrive ...
Sep 22 09:45:58 arch onedrive[814]: Sync with Microsoft OneDrive is complete

And even during the verification process, the modified file was uploaded.

I tested on versions 2.5.0 and onedrive v2.5.0-8-ge275d8d

abraunegg commented 3 days ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.