search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
9.97k stars 859 forks source link

Bug: Synchorinze is not working on subfolder #2696

Closed MuelNova closed 5 months ago

MuelNova commented 5 months ago

Describe the bug

when trying to synchronize /NovaNo1r/Blog folder, I found onedrive excluded NovaNo1r folder, so nothing is synchronized ( no folder created as well)

Operating System Details

Linux MuelNova-Arch 6.8.5-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Thu, 11 Apr 2024 01:47:18 +0000 x86_64 GNU/Linux

Client Installation Method

From Distribution Package

OneDrive Account Type

Business | Office365

What is your OneDrive Application Version

onedrive v2.4.25

What is your OneDrive Application Configuration

onedrive --confdir ~/.config/onedrive/accounts/n@ova.moe/ --display-config
Using config option for Global Azure AD Endpoints
Configuration file successfully loaded
onedrive version                             = v2.4.25
Config path                                  = /home/nova/.config/onedrive/accounts/n@ova.moe/
Config file found in config path             = true
Config option 'sync_dir'                     = /home/nova/OneDrive_n@ova.moe
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 5
Config option 'monitor_fullscan_frequency'   = 10
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = true
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = 
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   = 
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 125000000
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 600
Config option 'ip_protocol_version'          = 0
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = true
sync_list contents:
Blog/
NovaNo1r/Blog/
/NovaNo1r/Blog/
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = true
business_shared_folders contents:
Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 OpenSSL/3.2.1 zlib/1.3.1 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.7 libpsl/0.21.2 libssh2/1.11.0 nghttp2/1.61.0 nghttp3/1.2.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

mount                                                       
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=14220756k,nr_inodes=3555189,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p5 on / type btrfs (rw,noatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/@)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14907)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=14254192k,nr_inodes=1048576,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p5 on /home type btrfs (rw,noatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/@home)
/dev/nvme0n1p5 on /var/cache/pacman/pkg type btrfs (rw,noatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@pkg)
/dev/nvme0n1p5 on /var/log type btrfs (rw,noatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=258,subvol=/@log)
/dev/nvme0n1p6 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=2850836k,nr_inodes=712709,mode=700,uid=1000,gid=998,inode64)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=998)
kio-fuse on /run/user/1000/kio-fuse-eHRORf type fuse.kio-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=998)

What are all your local file system partition types

NAME        FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1                                                                                  
├─nvme0n1p1 vfat   FAT32 SYSTEM_DRV  2EB0-C2B1                               544M     8% /boot/efi
├─nvme0n1p2 ntfs         Windows-SSD 1A12B16C12B14D8F                                    
├─nvme0n1p3 ntfs         Data        A4860C8E860C62E2                                    
├─nvme0n1p4 ntfs         WINRE_DRV   D090B1E190B1CE6A                                    
├─nvme0n1p5 btrfs        Arch Linux  fac86794-4255-42e6-be01-6240e940bca7   46.4G    69% /var/log
│                                                                                        /var/cache/pacman/pkg
│                                                                                        /home
│                                                                                        /
└─nvme0n1p6 vfat   FAT32             D82C-37F3                             240.4M    62% /boot

How do you use 'onedrive'

When first synchronizing, it failed to synchronize.

Steps to reproduce the behaviour

Root file tree as shown:

.
├── Apps
├── Attachments
├── backup
├── CLASS
├── Noir @ novanoir
├── NovaNo1r
└── OneSyncFiles

under NovaNo1r folder

ree -L 1
.
├── some_pdf.pdf
├── ...
├── Blog
├── ...
└── other_stuffs.

Complete Verbose Log Output

Processing 202 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./NovaNo1r
Skipping item - excluded by sync_list config: ./Noir @ novanoir
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64RGDLL773SK65CJBQX7SZJ6BMN5
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64RB6N4SAV36O5CKUHQJ2ZXVYA7S
Skipping item - excluded by sync_list config: ./CLASS
Processing 206 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./.xdg-volume-info
Skipping item - excluded by sync_list config: ./NovaNo1r
Skipping item - excluded by sync_list config: ./CLASS
Processing 241 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./NovaNo1r
Skipping item - excluded by sync_list config: ./Noir @ novanoir
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64X46ZGMTHTIUVCJY6IKEHA5EUVE
Skipping item - excluded by sync_list config: ./CLASS
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64RNYBVHXWIFTRBIAMLMLLDTINT4
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64TOYQ5VRUZOQVF2GCUYBHMGPJSK
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64RYRFWOQ25UDZDLNIVFKDBH66TF
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64W53AOXUNLKUZDIMCWE7AJAWJRY
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64QR6B7D7Y7ZJRDKGUO2FZSJ4GC4
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64VRAVLUD7LPGVAYQXA2MXP6OPJN
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64Q4UXSYUJYLKJB3JF723VJ6ZPMS
WARNING: OneDrive API inconsistency - this file does not have any hash: 01FOPM64XCKXXAT72ZO5CYBJT3YPDFQRKQ
Processing 210 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./CLASS
Processing 241 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./CLASS
Processing 226 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./CLASS
Processing 220 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./CLASS
Processing 234 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./CLASS
Processing 247 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./CLASS
Processing 237 OneDrive items to ensure consistent local state due to sync_list being used
...

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 5 months ago

@MuelNova Firstly, this appears to be a very odd issue as this is explicitly tested in the ~150 use cases which are gone through when validating version releases, so I 100% suspect this has to do with how you are configuring your 'sync_list' file - thus this is not a bug - sorry.

A bug refers to technical issues like application crashes, memory leaks, or logical errors, warranting a GitHub issue for resolution. In contrast, difficulties with feature usage or integration challenges should prompt opening a GitHub discussion instead.

Please be mindful of this in the future.

As to your issue, from your config output, your 'sync_list' is the following:

Blog/
NovaNo1r/Blog/
/NovaNo1r/Blog/

I get what you are trying to achieve here - however you are not configuring your 'sync_list' correctly. The file excludes everything by default , however as you want to include specific directories, you will need specify to explicitly exclude the root directory, then include the sub directory:

# Include NovaNo1r/Blog 
# Parent and all other children of parent will be skipped
/NovaNo1r/Blog/*

Now - if the above suggestion does not resolve your issue , this will not be fixed in v2.4.25 - sorry.

My advice is that you change to start using v2.5.0-rc1 - please read https://github.com/abraunegg/onedrive/discussions/2662 on how to obtain, build and start using the RC1 client version which has been totally re-written.

If you still encounter an issue with v2.5.0-rc1 then this will be looked at before v2.5.0 release is done.

Please read:

abraunegg commented 5 months ago

@MuelNova I have confirmed the behavior with v2.4.25 and v2.5.0-rc1 and the behavior is as expected when using:

# Include NovaNo1r/Blog 
# Parent and all other children of parent will be skipped
/NovaNo1r/Blog/*

v2.4.25 handling

...
Processing 179 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by skip_dir config: random_25k_files
Performing a database consistency and integrity check on locally stored data ... 
Uploading differences of ~/OneDrive
Processing .
The directory has not changed
Processing random_data_s130
The directory has not changed
Uploading new items of ~/OneDrive
Skipping item - excluded by sync_list config: ./α
Skipping item - excluded by sync_list config: ./เอกสาร
Skipping item - excluded by skip_dir config: ./random_25k_files
OneDrive Client requested to create remote path: ./NovaNo1r
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r
Successfully created the remote directory ./NovaNo1r on OneDrive
Skipping item - excluded by sync_list config: ./NovaNo1r/some_pdf.pdf
OneDrive Client requested to create remote path: ./NovaNo1r/Blog
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog
Successfully created the remote directory ./NovaNo1r/Blog on OneDrive
OneDrive Client requested to create remote path: ./NovaNo1r/Blog/random_files
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_files
Successfully created the remote directory ./NovaNo1r/Blog/random_files on OneDrive
OneDrive Client requested to create remote path: ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y
Successfully created the remote directory ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y on OneDrive
Uploading new file ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y/file0.data ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:03                                                                                                    
done.
Remaining free space on OneDrive: 977826614671
...

v2.5.0 handling

...
Skipping item - excluded by sync_list config: random_25k_files/random_files/01cCMOsxq81xCMnrWzt1fDpNx0Ieoqt2/file107.data
Skipping item - excluded by sync_list config: random_25k_files/random_files/01cCMOsxq81xCMnrWzt1fDpNx0Ieoqt2/file171.data
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Processing: ~/OneDrive
The directory has not changed
Processing: random_data_s130
The directory has not changed
Scanning the local file system '~/OneDrive' for new data to upload
Skipping item - excluded by sync_list config: ./α
Skipping item - excluded by sync_list config: ./เอกสาร
Skipping item - excluded by skip_dir config: ./random_25k_files
OneDrive Client requested to create this directory online: ./NovaNo1r
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r
Successfully created the remote directory ./NovaNo1r on Microsoft OneDrive
Skipping item - excluded by sync_list config: ./NovaNo1r/some_pdf.pdf
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog
Successfully created the remote directory ./NovaNo1r/Blog on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog/random_files
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_files
Successfully created the remote directory ./NovaNo1r/Blog/random_files on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y
Successfully created the remote directory ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog/random_files/CZ9aZRM7U1j7pM21fH0MfP2gywlX7bqW
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_files/CZ9aZRM7U1j7pM21fH0MfP2gywlX7bqW
Successfully created the remote directory ./NovaNo1r/Blog/random_files/CZ9aZRM7U1j7pM21fH0MfP2gywlX7bqW on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog/random_images
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_images
Successfully created the remote directory ./NovaNo1r/Blog/random_images on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog/random_images/cAuQMfX7qsMIOmzyQYdELikZwsXeCYsL
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_images/cAuQMfX7qsMIOmzyQYdELikZwsXeCYsL
Successfully created the remote directory ./NovaNo1r/Blog/random_images/cAuQMfX7qsMIOmzyQYdELikZwsXeCYsL on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./NovaNo1r/Blog/random_images/GqjZuo7UBB0qjYM2WUcZXOvToAhCQ29M
The requested directory to create was not found on OneDrive - creating remote directory: ./NovaNo1r/Blog/random_images/GqjZuo7UBB0qjYM2WUcZXOvToAhCQ29M
Successfully created the remote directory ./NovaNo1r/Blog/random_images/GqjZuo7UBB0qjYM2WUcZXOvToAhCQ29M on Microsoft OneDrive
Skipping item - excluded by sync_list config: ./NovaNo1r/other_stuffs
New items to upload to OneDrive: 20
Total New Data to Upload:        77 MB
Uploading: ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y/file0.data ... 0%   |  ETA    --:--:--
Uploading: ./NovaNo1r/Blog/random_files/k4GptfTBE2z2meRFqjf54tnvSXcXe30Y/file1.data ... 0%   |  ETA    --:--:--
...

I have also updated the v2.5.x documentation to ensure that this sort of configuration is clearer to understand.

As such, I am closing this issue as this is not a bug (as demonstrated) but is a client configuration issue and partly a documentation issue which has been fixed.

MuelNova commented 5 months ago

Thank you for your detailed answer. I have tested that and everything works well right now.😄

However, there are something I want to explain though.

I do see that "everything will be excluded", but what I have wrote was 

/NovaNo1r
!/NovaNo1r/*
/NovaNo1r/Blog

Which, from now perspective, is kinda silly.

Thank you for your kind advice. I know that I should post on GitHub discussion seeking for the help of advanced usage. But what I was thinking is that this is a bug, because I clearly remember that I configured like this before, and they worked as expected when I was using VM. So I guess it could be a bug related to the distribution or file system (maybe I just accidentally configured correctly)

I will make a deeper investigation next time before I open a GitHub issue.

Thank you again for making this amazing project and for your detailed answer.

abraunegg commented 5 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.