search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
9.63k stars 849 forks source link

Bug: Cannot select/filter within Personal Shared Folders #2731

Open robertschulze opened 1 month ago

robertschulze commented 1 month ago

Describe the bug

Files/folders from a Personal Shared Folder cannot be filtered because the entries are not considered with their full paths. I.e. the contents of sync_list create no match.

Consider the following example setup: OneDrive Root --> Test 1 (actual folder) OneDrive Root --> OneDrive - Constanze (shared folder from another account)

The sync_list is as follows (%20 entries are necessary because I cannot get rid of the url-encoded spaces, cf. #2562)

/OneDrive - Constanze/*                                                         
/OneDrive%20-%20Constanze/*                                                     
/Test 1/*                                                                       
/Test%201/*

The relevant part of the log is:

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 3 of 3 as part of JSON Item Batch 1 of 1
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"aYzo5QkE2RjA3NjY4QkExRkY3ITIyMzg3LjI1OA","eTag":"aOUJBNkYwNzY2OEJBMUZGNyEyMjM4Ny4z","file":{"hashes":{"quickXorHash":"YZgDGcwAAAAAAAAABAAAAAAAAAA=","sha1Hash":"3DA541559918A808C2402BBA5012F6C60B27661C","sha256Hash":"F0E4C2F76C58916EC258F246851BEA091D14D4247A2FC3E18694461B1816E13B"},"mimeType":"text\/plain"},"fileSystemInfo":{"createdDateTime":"2024-05-31T21:32:04.763Z","lastModifiedDateTime":"2024-05-31T21:32:27Z"},"id":"9BA6F07668BA1FF7!22387","name":"test1doc.txt","parentReference":{"driveId":"9ba6f07668ba1ff7","driveType":"personal","id":"9BA6F07668BA1FF7!22385","name":"Test 1","path":"\/drive\/root:\/Test%201"},"size":4}
DEBUG: Attempting to calculate local filesystem path for 9ba6f07668ba1ff7 and 9BA6F07668BA1FF7!22385
DEBUG: JSON Item calculated full path is: Test 1/test1doc.txt
DEBUG: The item we are syncing is a file
DEBUG: skip_file item to check: /Test 1/test1doc.txt
DEBUG: skip_file evaluation for: /Test 1/test1doc.txt
DEBUG: Result: false
DEBUG: sync_list item to check: Test 1/test1doc.txt
DEBUG: Evaluation against 'sync_list' for this path: Test 1/test1doc.txt
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /Test 1/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /Test%201/*
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a file
DEBUG: OneDrive change is potentially a new local item
DEBUG: Path on local disk already exists
DEBUG: Local file time discrepancy detected: Test 1/test1doc.txt
DEBUG: This local file has a different modified time 2024-May-31 22:21:54Z (UTC) when compared to remote modified time 2024-May-31 21:32:27Z (UTC)
DEBUG: Local item has the same hash value as the item online - correcting the applicable file timestamp
DEBUG: The source of the incorrect timestamp was the local file - correcting timestamp locally due to --download-only
DEBUG: Calling setTimes() for this file: Test 1/test1doc.txt
DEBUG: The item to sync exists locally but is potentially not in the local database - otherwise this would be handled as changed item
DEBUG: File timestamps are equal, no further action required
DEBUG: Update/Insert local database with item details: Item("9ba6f07668ba1ff7", "9BA6F07668BA1FF7!22387", "test1doc.txt", "", file, "aOUJBNkYwNzY2OEJBMUZGNyEyMjM4Ny4z", "aYzo5QkE2RjA3NjY4QkExRkY3ITIyMzg3LjI1OA", 2024-May-31 21:32:27Z, "9BA6F07668BA1FF7!22385", "YZgDGcwAAAAAAAAABAAAAAAAAAA=", "", "", "", "", none, "Y", "4")
DEBUG: Batched JSON item processing time: 345 ms, 935 μs, and 1 hnsec
DEBUG: ------------------------------------------------------------------
DEBUG: Syncing this OneDrive Personal Shared Folder: OneDrive - Constanze
DEBUG: Why are we generating a /delta response:
DEBUG:  singleDirectoryScope:    false
DEBUG:  nationalCloudDeployment: false
DEBUG:  cleanupLocalFiles:       true
DEBUG: CurlEngine getCurlInstance() called
DEBUG: CurlEngine curlEnginePool current size: 3
DEBUG: CurlEngine was in a valid state - returning existing CurlEngine instance
DEBUG: CurlEngine instance ID: HRAP4XTgEzLNoBfl
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: Request URL = https://graph.microsoft.com/v1.0/me/drive/root:/OneDrive%20-%20Constanze:/?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-Jun-01 01:21:53.3723092
DEBUG: HTTP Response Headers: ["strict-transport-security":"max-age=31536000", "x-ms-ags-diagnostic":"{\"ServerInfo\":{\"DataCenter\":\"Germany West Central\",\"Slice\":\"E\",\"Ring\":\"4\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"FR2PEPF000001E9\"}}", "odata-version":"4.0", "date":"Fri, 31 May 2024 22:21:54 GMT", "client-request-id":"6a37e3e3-ba9a-41a3-943d-f065d86740a5", "request-id":"6a37e3e3-ba9a-41a3-943d-f065d86740a5", "cache-control":"private", "content-type":"application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"]
DEBUG: HTTP Status Line: 200  (2.0)
DEBUG: CurlEngine cleanup() called on instance id: HRAP4XTgEzLNoBfl
DEBUG: Downgrading all children for this searchItem.driveId (f3028bd758552faf) and searchItem.id (F3028BD758552FAF!4690) to an out-of-sync state
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/f3028bd758552faf/items/F3028BD758552FAF!4690?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-Jun-01 01:21:53.3723092
DEBUG: HTTP Response Headers: ["strict-transport-security":"max-age=31536000", "x-ms-ags-diagnostic":"{\"ServerInfo\":{\"DataCenter\":\"Germany West Central\",\"Slice\":\"E\",\"Ring\":\"4\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"FR2PEPF000001E9\"}}", "odata-version":"4.0", "date":"Fri, 31 May 2024 22:21:54 GMT", "client-request-id":"afe5320a-8d04-4deb-981d-7f93723c60d6", "request-id":"afe5320a-8d04-4deb-981d-7f93723c60d6", "cache-control":"private", "content-type":"application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"]
DEBUG: HTTP Status Line: 200  (2.0)
DEBUG: CurlEngine cleanup() called on instance id: HRAP4XTgEzLNoBfl
DEBUG: Generating a /delta response from the OneDrive API for Drive ID: f3028bd758552faf
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/f3028bd758552faf/root
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-Jun-01 01:21:53.3723092
DEBUG: HTTP Response Headers: ["strict-transport-security":"max-age=31536000", "x-ms-ags-diagnostic":"{\"ServerInfo\":{\"DataCenter\":\"Germany West Central\",\"Slice\":\"E\",\"Ring\":\"4\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"FR2PEPF000001E9\"}}", "odata-version":"4.0", "date":"Fri, 31 May 2024 22:21:55 GMT", "client-request-id":"945efaff-e09d-4d67-9cb5-c17e15e442ba", "request-id":"945efaff-e09d-4d67-9cb5-c17e15e442ba", "cache-control":"no-store", "content-type":"application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"]
DEBUG: HTTP Status Line: 200  (2.0)
DEBUG: CurlEngine cleanup() called on instance id: HRAP4XTgEzLNoBfl
DEBUG: Adding OneDrive root details for processing
DEBUG: Adding OneDrive folder details for processing
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/f3028bd758552faf/items/F3028BD758552FAF!4690/children?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-Jun-01 01:21:53.3723092
DEBUG: HTTP Response Headers: ["strict-transport-security":"max-age=31536000", "x-ms-ags-diagnostic":"{\"ServerInfo\":{\"DataCenter\":\"Germany West Central\",\"Slice\":\"E\",\"Ring\":\"4\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"FR2PEPF000001E9\"}}", "odata-version":"4.0", "date":"Fri, 31 May 2024 22:21:56 GMT", "client-request-id":"e6db94bd-78d3-4a46-86bb-065fec836c7b", "request-id":"e6db94bd-78d3-4a46-86bb-065fec836c7b", "cache-control":"no-store", "content-type":"application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"]
DEBUG: HTTP Status Line: 200  (2.0)
DEBUG: CurlEngine cleanup() called on instance id: HRAP4XTgEzLNoBfl
DEBUG: Adding 13 OneDrive items for processing from the OneDrive 'root' folder
DEBUG: sync_list item to check: Basteln
DEBUG: Evaluation against 'sync_list' for this path: Basteln
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /Test 1/*
DEBUG: Evaluation against 'sync_list' entry: /Test%201/*
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: EXCLUDED
DEBUG: Skipping item - excluded by sync_list config: Basteln

So unfortunately the folder Basteln from OneDrive - Constanze is not synced while a test file within Test 1 folder is synced properly. The expected behaviour would be to instead of

DEBUG: Evaluation against 'sync_list' for this path: Basteln

have:

DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Constanze/Basteln

FYI: I was able to solve #2620 by upgrading smbclient to 4.19.5 (manual compile in Ubuntu). Unfortunately, I cannot write the information to #2620 because it is already closed.

Operating System Details

Linux atom 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.0-rc2-25-g71bce90

What is your OneDrive Application Configuration

onedrive version                             = onedrive v2.5.0-rc2-25-g71bce90
Config path                                  = /home/robert/.config/onedrive
Config file found in config path             = false
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc2-25-g71bce90
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.17
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Network

What are all your system 'mount points'

//hyperv/3 on /media/3 type cifs (rw,relatime,vers=3.1.1,cache=strict,username=Administrator,uid=1000,noforceuid,gid=100,noforcegid,addr=192.168.3.11,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

What are all your local file system partition types

NAME                  FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                 squashfs    4.0                                                         0   100% /snap/bare/5
loop1                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/264
loop2                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/278
loop3                 squashfs    4.0                                                         0   100% /snap/core/16574
loop4                 squashfs    4.0                                                         0   100% /snap/core/16928
loop5                 squashfs    4.0                                                         0   100% /snap/core18/2812
loop6                 squashfs    4.0                                                         0   100% /snap/core18/2823
loop7                 squashfs    4.0                                                         0   100% /snap/core20/2264
loop8                 squashfs    4.0                                                         0   100% /snap/core20/2318
loop9                 squashfs    4.0                                                         0   100% /snap/core22/1122
loop10                squashfs    4.0                                                         0   100% /snap/core22/1380
loop11                squashfs    4.0                                                         0   100% /snap/cups/1044
loop12                squashfs    4.0                                                         0   100% /snap/cups/1047
loop14                squashfs    4.0                                                         0   100% /snap/firefox/4259
loop15                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/104
loop16                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/111
loop17                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/194
loop18                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/198
loop19                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/90
loop20                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/93
loop21                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/140
loop22                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/143
loop23                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/172
loop24                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/176
loop25                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/184
loop26                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/186
loop27                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1534
loop28                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1535
loop29                squashfs    4.0                                                         0   100% /snap/snap-store/1113
loop30                squashfs    4.0                                                         0   100% /snap/snap-store/959
loop31                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/157
loop32                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/83
loop33                                                                                        0   100% /snap/firefox/4336
sda                                                                                                    
├─sda1                vfat        FAT32          62EB-8E64                               503,3M     1% /boot/efi
├─sda2                                                                                                 
└─sda5                LVM2_member LVM2 001       X34xd2-HOp4-sAHk-pudA-CzVG-rENF-FOtrO2                
  ├─ubuntu--vg-root   ext4        1.0            43f5fc57-8370-4315-b5ba-95912f21e113     20,8G    74% /var/snap/firefox/common/host-hunspell
  │                                                                                                    /media/robert_home
  │                                                                                                    /
  └─ubuntu--vg-swap_1 swap        1              ad32bbec-507e-44b0-bb5f-1df3ee647a2a                  [SWAP]

How do you use 'onedrive'

cf. main bug description above

Steps to reproduce the behaviour

./onedrive --confdir='/home/robert/.config/onedrive/accounts/robert@guitaronline.de' --sync --verbose --verbose --resync --resync-auth > debug_output.log 2>&1

Complete Verbose Log Output

Can provide via mail in case needed.

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 1 month ago

@robertschulze Please can you email a full, unredacted debug log for analysis.

Unfortunately I will not be able to look at this is short term as I am in the middle of doing some re-work at the moment. Post doing that, this will be looked at.