Bug: HTML encoding requires to add filter entries twice

robertschulze commented 1 month ago

Describe the bug

In my sync_list I have (among others):

/OneDrive - Elias/*                                                                                                                                                         
/OneDrive%20-%20Elias/*                                                                                                                                                     
Backup                                                                                                                                                                      
Bücher

To sync subdirs "Backup" and "Bücher" of folder "OneDrive - Elias". However, directory "Bücher" is not entered, rather the seek continues in "Dokumente"

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 333 of 500 as part of JSON Item Batch 1 of 2
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITI0MDYuNjM4NTM5ODU4NjM1NTAwMDAw","eTag":"aRTcyNDcwMTZDNzgyNkZCQyEyNDA2LjA","fileSystemInfo":{"createdDateTime":"2024-04-24T18:58:17.283Z","lastModifiedDateTime":"2024-04-24T18:58:13Z"},"folder":{"childCount":25,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!2406","name":"Bücher","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":2816411537}
DEBUG: Attempting to calculate local filesystem path for e7247016c7826fbc and E7247016C7826FBC!103
DEBUG: JSON Item calculated full path is: OneDrive - Elias/Bücher
DEBUG: The item we are syncing is a folder
DEBUG: sync_list item to check: OneDrive - Elias/Bücher
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Elias/Bücher
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Creating local directory: ./OneDrive - Elias/Bücher
DEBUG: Requested path does not exist, creating directory structure: OneDrive - Elias/Bücher
DEBUG: Setting directory permissions for: OneDrive - Elias/Bücher
DEBUG: Setting directory lastModifiedDateTime for: OneDrive - Elias/Bücher to 2024-Apr-24 18:58:13Z
DEBUG: Calling setTimes() for this directory: OneDrive - Elias/Bücher
DEBUG: saveItem - creating DB item from this JSON: {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITI0MDYuNjM4NTM5ODU4NjM1NTAwMDAw","eTag":"aRTcyNDcwMTZDNzgyNkZCQyEyNDA2LjA","fileSystemInfo":{"createdDateTime":"2024-04-24T18:58:17.283Z","lastModifiedDateTime":"2024-04-24T18:58:13Z"},"folder":{"childCount":25,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!2406","name":"Bücher","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":2816411537}
DEBUG: Flagging object as a directory
DEBUG: Batched JSON item processing time: 2 ms, 502 μs, and 5 hnsecs
DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 334 of 500 as part of JSON Item Batch 1 of 2
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITE2Ni42Mzg1Mjc0MDc3NTA5MzAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExNjYuMA","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:42.89Z","lastModifiedDateTime":"2022-12-18T13:12:37Z"},"folder":{"childCount":15,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!166","name":"Dokumente","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":2463584284}
DEBUG: Attempting to calculate local filesystem path for e7247016c7826fbc and E7247016C7826FBC!103
DEBUG: JSON Item calculated full path is: OneDrive - Elias/Dokumente
DEBUG: The item we are syncing is a folder
DEBUG: sync_list item to check: OneDrive - Elias/Dokumente
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Elias/Dokumente
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Creating local directory: ./OneDrive - Elias/Dokumente
DEBUG: Requested path does not exist, creating directory structure: OneDrive - Elias/Dokumente
DEBUG: Setting directory permissions for: OneDrive - Elias/Dokumente
DEBUG: Setting directory lastModifiedDateTime for: OneDrive - Elias/Dokumente to 2022-Dec-18 13:12:37Z
DEBUG: Calling setTimes() for this directory: OneDrive - Elias/Dokumente
DEBUG: saveItem - creating DB item from this JSON: {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITE2Ni42Mzg1Mjc0MDc3NTA5MzAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExNjYuMA","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:42.89Z","lastModifiedDateTime":"2022-12-18T13:12:37Z"},"folder":{"childCount":15,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!166","name":"Dokumente","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":2463584284}
DEBUG: Flagging object as a directory
DEBUG: Batched JSON item processing time: 2 ms, 913 μs, and 1 hnsec
DEBUG: ------------------------------------------------------------------

while for the parallel directory "Backup", the subdirectory "Backup/20221026 - Kindle" is considered

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 159 of 500 as part of JSON Item Batch 1 of 2
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITEyMS42MzgzMDE2MzA5MDI0MzAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExMjEuMA","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:20.583Z","lastModifiedDateTime":"2022-12-18T13:12:15Z"},"folder":{"childCount":1,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!121","name":"Backup","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":278981479}
DEBUG: Attempting to calculate local filesystem path for e7247016c7826fbc and E7247016C7826FBC!103
DEBUG: JSON Item calculated full path is: OneDrive - Elias/Backup
DEBUG: The item we are syncing is a folder
DEBUG: sync_list item to check: OneDrive - Elias/Backup
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Elias/Backup
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Creating local directory: ./OneDrive - Elias/Backup
DEBUG: Requested path does not exist, creating directory structure: OneDrive - Elias/Backup
DEBUG: Setting directory permissions for: OneDrive - Elias/Backup
DEBUG: Setting directory lastModifiedDateTime for: OneDrive - Elias/Backup to 2022-Dec-18 13:12:15Z
DEBUG: Calling setTimes() for this directory: OneDrive - Elias/Backup
DEBUG: saveItem - creating DB item from this JSON: {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITEyMS42MzgzMDE2MzA5MDI0MzAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExMjEuMA","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:20.583Z","lastModifiedDateTime":"2022-12-18T13:12:15Z"},"folder":{"childCount":1,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!121","name":"Backup","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":278981479}
DEBUG: Flagging object as a directory
DEBUG: Batched JSON item processing time: 28 ms, 142 μs, and 1 hnsec
DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 160 of 500 as part of JSON Item Batch 1 of 2
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITEyMi42MzgzMDE2MzA5MDI0MzAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExMjIuMA","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:20.886Z","lastModifiedDateTime":"2022-12-18T13:12:15Z"},"folder":{"childCount":9,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!122","name":"20221026 - Kindle","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!121","name":"Backup","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:\/Backup"},"size":278981479}
DEBUG: Attempting to calculate local filesystem path for e7247016c7826fbc and E7247016C7826FBC!121
DEBUG: JSON Item calculated full path is: OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: The item we are syncing is a folder
DEBUG: sync_list item to check: OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Creating local directory: ./OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: Requested path does not exist, creating directory structure: OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: Setting directory permissions for: OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: Setting directory lastModifiedDateTime for: OneDrive - Elias/Backup/20221026 - Kindle to 2022-Dec-18 13:12:15Z
DEBUG: Calling setTimes() for this directory: OneDrive - Elias/Backup/20221026 - Kindle
DEBUG: saveItem - creating DB item from this JSON: {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITEyMi42MzgzMDE2MzA5MDI0MzAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExMjIuMA","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:20.886Z","lastModifiedDateTime":"2022-12-18T13:12:15Z"},"folder":{"childCount":9,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!122","name":"20221026 - Kindle","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!121","name":"Backup","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:\/Backup"},"size":278981479}
DEBUG: Flagging object as a directory
DEBUG: Batched JSON item processing time: 16 ms, 471 μs, and 9 hnsecs
DEBUG: ------------------------------------------------------------------

Further down in the log file it turns out the subdir "20181006 - Fredo 1 Mathematik Arbeitsheft" was excluded

DEBUG: Adding 25 OneDrive items for processing from Bücher
DEBUG: CAUTION:    The JSON element transmitted by the Microsoft OneDrive API includes HTML URL encoded items, which may complicate pattern matching and potentially lead to synchronisation problems for this item.
DEBUG: WORKAROUND: An alternative solution could be to change the name of this item through the online platform: /B%C3%BCcher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: See: https://github.com/OneDrive/onedrive-api-docs/issues/1765 for further details
DEBUG: sync_list item to check: B%C3%BCcher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: Evaluation against 'sync_list' for this path: B%C3%BCcher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: EXCLUDED
DEBUG: Skipping item - excluded by sync_list config: B%C3%BCcher/20181006 - Fredo 1 Mathematik Arbeitsheft

Only when now adding "B%C3%BCcher" to the sync_list as well, the files and folders within "Bücher" are properly synchronized:

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 333 of 500 as part of JSON Item Batch 1 of 3
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITI0MDYuNjM4NTM5ODU4NjM1NTAwMDAw","eTag":"aRTcyNDcwMTZDNzgyNkZCQyEyNDA2LjA","fileSystemInfo":{"createdDateTime":"2024-04-24T18:58:17.283Z","lastModifiedDateTime":"2024-04-24T18:58:13Z"},"folder":{"childCount":25,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!2406","name":"Bücher","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:"},"size":2816411537}
DEBUG: Attempting to calculate local filesystem path for e7247016c7826fbc and E7247016C7826FBC!103
DEBUG: JSON Item calculated full path is: OneDrive - Elias/Bücher
DEBUG: The item we are syncing is a folder
DEBUG: sync_list item to check: OneDrive - Elias/Bücher
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Elias/Bücher
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: Evaluation against 'sync_list' entry: B%C3%BCcher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Path on local disk already exists
DEBUG: The item to sync is already present on the local filesystem and is in-sync with what is reported online
DEBUG: Update/Insert local database with item details: Item("e7247016c7826fbc", "E7247016C7826FBC!2406", "Bücher", "", dir, "aRTcyNDcwMTZDNzgyNkZCQyEyNDA2LjA", "adDpFNzI0NzAxNkM3ODI2RkJDITI0MDYuNjM4NTM5ODU4NjM1NTAwMDAw", 2024-Apr-24 18:58:13Z, "E7247016C7826FBC!103", "", "", "", "", "", none, "Y", "")
DEBUG: Batched JSON item processing time: 476 μs and 9 hnsecs
DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 334 of 500 as part of JSON Item Batch 1 of 3
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITE4OS42Mzg0OTU4MTkxMDM5MDAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExODkuMQ","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:44.993Z","lastModifiedDateTime":"2019-07-03T16:00:46Z"},"folder":{"childCount":32,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!189","name":"20181006 - Fredo 1 Mathematik Arbeitsheft","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!2406","name":"Bücher","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:\/B%C3%BCcher"},"size":7104281}
DEBUG: Attempting to calculate local filesystem path for e7247016c7826fbc and E7247016C7826FBC!2406
DEBUG: JSON Item calculated full path is: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: The item we are syncing is a folder
DEBUG: sync_list item to check: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Constanze/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Constanze/*
DEBUG: Evaluation against 'sync_list' entry: Basteln
DEBUG: Evaluation against 'sync_list' entry: Datenbanken
DEBUG: Evaluation against 'sync_list' entry: Dokumente
DEBUG: Evaluation against 'sync_list' entry: Fotos
DEBUG: Evaluation against 'sync_list' entry: Gesundheit
DEBUG: Evaluation against 'sync_list' entry: Logo
DEBUG: Evaluation against 'sync_list' entry: Musik
DEBUG: Evaluation against 'sync_list' entry: Privat
DEBUG: Evaluation against 'sync_list' entry: Schule
DEBUG: Evaluation against 'sync_list' entry: Studium
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Rahel/*
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Rahel/*
DEBUG: Evaluation against 'sync_list' entry: Artwork
DEBUG: Evaluation against 'sync_list' entry: Freunde
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: /OneDrive%20-%20Elias/*
DEBUG: Evaluation against 'sync_list' entry: Backup
DEBUG: Evaluation against 'sync_list' entry: Bücher
DEBUG: Evaluation against 'sync_list' entry: B%C3%BCcher
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Creating local directory: ./OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: Requested path does not exist, creating directory structure: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: Setting directory permissions for: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: Setting directory lastModifiedDateTime for: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft to 2019-Jul-03 16:00:46Z
DEBUG: Calling setTimes() for this directory: OneDrive - Elias/Bücher/20181006 - Fredo 1 Mathematik Arbeitsheft
DEBUG: saveItem - creating DB item from this JSON: {"cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITE4OS42Mzg0OTU4MTkxMDM5MDAwMDA","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExODkuMQ","fileSystemInfo":{"createdDateTime":"2022-12-18T13:12:44.993Z","lastModifiedDateTime":"2019-07-03T16:00:46Z"},"folder":{"childCount":32,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"E7247016C7826FBC!189","name":"20181006 - Fredo 1 Mathematik Arbeitsheft","parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!2406","name":"Bücher","path":"\/drives\/e7247016c7826fbc\/items\/E7247016C7826FBC!103:\/B%C3%BCcher"},"size":7104281}
DEBUG: Flagging object as a directory
DEBUG: Batched JSON item processing time: 3 ms, 295 μs, and 1 hnsec
DEBUG: ------------------------------------------------------------------

Operating System Details

Linux atom 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.0-rc2-37-g3f7fb5a

What is your OneDrive Application Configuration

onedrive version                             = onedrive v2.5.0-rc2-37-g3f7fb5a
Config path                                  = /home/robert/.config/onedrive
Config file found in config path             = false
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc2-37-g3f7fb5a
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 8.8.0 (x86_64-pc-linux-gnu) libcurl/8.8.0 OpenSSL/3.0.2 zlib/1.2.11 libidn2/2.3.2 nghttp2/1.43.0 OpenLDAP/2.5.17
Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

Where is your 'sync_dir' located

Network

What are all your system 'mount points'

//hyperv/3 on /media/3 type cifs (rw,relatime,vers=3.1.1,cache=strict,username=Administrator,uid=1000,noforceuid,gid=100,noforcegid,addr=192.168.3.11,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

What are all your local file system partition types

NAME                  FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                 squashfs    4.0                                                         0   100% /snap/bare/5
loop1                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/264
loop2                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/278
loop3                 squashfs    4.0                                                         0   100% /snap/core/16574
loop4                 squashfs    4.0                                                         0   100% /snap/core/16928
loop5                 squashfs    4.0                                                         0   100% /snap/core18/2812
loop6                 squashfs    4.0                                                         0   100% /snap/core18/2823
loop7                 squashfs    4.0                                                         0   100% /snap/core20/2264
loop8                 squashfs    4.0                                                         0   100% /snap/core20/2318
loop9                 squashfs    4.0                                                         0   100% /snap/core22/1122
loop10                squashfs    4.0                                                         0   100% /snap/core22/1380
loop11                squashfs    4.0                                                         0   100% /snap/cups/1044
loop12                squashfs    4.0                                                         0   100% /snap/cups/1047
loop14                squashfs    4.0                                                         0   100% /snap/firefox/4259
loop15                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/104
loop16                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/111
loop17                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/194
loop18                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/198
loop19                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/90
loop20                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/93
loop21                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/140
loop22                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/143
loop23                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/172
loop24                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/176
loop25                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/184
loop26                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/186
loop27                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1534
loop28                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1535
loop29                squashfs    4.0                                                         0   100% /snap/snap-store/1113
loop30                squashfs    4.0                                                         0   100% /snap/snap-store/959
loop31                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/157
loop32                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/83
loop33                                                                                        0   100% /snap/firefox/4336
sda                                                                                                    
├─sda1                vfat        FAT32          62EB-8E64                               503,3M     1% /boot/efi
├─sda2                                                                                                 
└─sda5                LVM2_member LVM2 001       X34xd2-HOp4-sAHk-pudA-CzVG-rENF-FOtrO2                
  ├─ubuntu--vg-root   ext4        1.0            43f5fc57-8370-4315-b5ba-95912f21e113     20,8G    74% /var/snap/firefox/common/host-hunspell
  │                                                                                                    /media/robert_home
  │                                                                                                    /
  └─ubuntu--vg-swap_1 swap        1              ad32bbec-507e-44b0-bb5f-1df3ee647a2a                  [SWAP]

How do you use 'onedrive'

./onedrive --confdir='/home/robert/.config/onedrive/accounts/robert@guitaronline.de' --sync --verbose --verbose --resync --resync-auth > debug_output.log 2>&1

Steps to reproduce the behaviour

see description above

Complete Verbose Log Output

see description above, can send via mail if required

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 1 month ago

@robertschulze

HTML encoding requires to add filter entries twice

This is not a bug.

For whatever reason - you are the only person who seems to have paths stored online that use HTML entities - I am yet to even find a way locally or remotely that creates folders that contain things like '%20'. How are these being even created in the first place or is this some sort of 'legacy' naming issue ?

The folder entity that is returned from Microsoft OneDrive includes the HTML entity itself in the JSON response - thus, to filter against that specific folder you need to add the folder name that contains that entity.

Your solution here is to really rename your folders .. failing that it is double entries .. sorry.

robertschulze commented 1 month ago

Very strange, I don't think I did anything special to create these. Maybe the HTML entities are an artifact of me "mounting" other OneDrives within my main OneDrive?

abraunegg commented 1 month ago

Very strange, I don't think I did anything special to create these. Maybe the HTML entities are an artifact of me "mounting" other OneDrives within my main OneDrive?

No, that does not create that effect, neither can you create a folder or link with %20 - as when I tested this last time you brought this up - OneDrive prevents the creation... so this has to be something with how you are doing things online.

robertschulze commented 1 month ago

You are probably right: even if I create a folder "BücherTest" within my main OneDrive (not a linked one) in the main root dir the special letter is HTML encoded even in the browser: Maybe it is due to the fact that I am using OneDrive in German?

robertschulze commented 1 month ago

For reference I also tested on a corporate SharePoint and also here folders with special characters are HTML encoded: For pure English locale this is not the case? Or maybe I am checking in the wrong place?

abraunegg commented 1 month ago

@robertschulze What should be happening is that UTF-16 encoding should be used - for example:

drwx------. 1 alex alex   0 Feb  8 06:57  α
drwx------. 1 alex alex   0 Feb  8 06:57  เอกสาร

These are stored online correctly as UTF-16, for example:

You also have some sort of DecodePath item also being 'included' .. which is not occurring for any item I am using:

So I created a new folder online called Bücher using the text from this issue ticket, and when I look at the verbose debug logging (I have to use a test personal account here to demonstrate as I am doing long running memory tests using Business|Sharepoint)

DEBUG: Processing OneDrive Item 2 of 2 from API Response Bundle 1
DEBUG: Raw JSON OneDrive Item: {"cTag":"adDo2NkQ1M0JFOEE1MDU2RUNBITQzMjE1OC42Mzg1NDE4MjMwMjc1MDAwMDA","eTag":"aNjZENTNCRThBNTA1NkVDQSE0MzIxNTguMg","fileSystemInfo":{"createdDateTime":"2024-06-17T00:40:15.18Z","lastModifiedDateTime":"2024-06-17T00:40:15.18Z"},"folder":{"childCount":0,"view":{"sortBy":"takenOrCreatedDateTime","sortOrder":"ascending","viewType":"thumbnails"}},"id":"66D53BE8A5056ECA!432158","name":"Bücher","parentReference":{"driveId":"66d53be8a5056eca","driveType":"personal","id":"66D53BE8A5056ECA!101","name":"root:","path":"\/drive\/root:"},"size":0}

The 'name' as provided by Microsoft OneDrive API includes the umlaut ... not some HTML entity version of the text. The JSON is processed 100% normally:

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 1 of 1 as part of JSON Item Batch 1 of 1
DEBUG: Raw JSON OneDrive Item (Batched Item): {"cTag":"adDo2NkQ1M0JFOEE1MDU2RUNBITQzMjE1OC42Mzg1NDE4MjMwMjc1MDAwMDA","eTag":"aNjZENTNCRThBNTA1NkVDQSE0MzIxNTguMg","fileSystemInfo":{"createdDateTime":"2024-06-17T00:40:15.18Z","lastModifiedDateTime":"2024-06-17T00:40:15.18Z"},"folder":{"childCount":0,"view":{"sortBy":"takenOrCreatedDateTime","sortOrder":"ascending","viewType":"thumbnails"}},"id":"66D53BE8A5056ECA!432158","name":"Bücher","parentReference":{"driveId":"66d53be8a5056eca","driveType":"personal","id":"66D53BE8A5056ECA!101","name":"root:","path":"\/drive\/root:"},"size":0}
DEBUG: Attempting to calculate local filesystem path for 66d53be8a5056eca and 66D53BE8A5056ECA!101
DEBUG: JSON Item calculated full path is: ./Bücher
DEBUG: The item we are syncing is a folder
DEBUG: Flagging object as a directory
DEBUG: OneDrive change is potentially a new local item
DEBUG: Creating local directory: ./Bücher
DEBUG: Requested path does not exist, creating directory structure: ./Bücher
DEBUG: Setting directory permissions for: ./Bücher
DEBUG: Setting directory lastModifiedDateTime for: ./Bücher to 2024-Jun-17 00:40:15.18Z
DEBUG: Calling setTimes() for this directory: ./Bücher
DEBUG: saveItem - creating DB item from this JSON: {"cTag":"adDo2NkQ1M0JFOEE1MDU2RUNBITQzMjE1OC42Mzg1NDE4MjMwMjc1MDAwMDA","eTag":"aNjZENTNCRThBNTA1NkVDQSE0MzIxNTguMg","fileSystemInfo":{"createdDateTime":"2024-06-17T00:40:15.18Z","lastModifiedDateTime":"2024-06-17T00:40:15.18Z"},"folder":{"childCount":0,"view":{"sortBy":"takenOrCreatedDateTime","sortOrder":"ascending","viewType":"thumbnails"}},"id":"66D53BE8A5056ECA!432158","name":"Bücher","parentReference":{"driveId":"66d53be8a5056eca","driveType":"personal","id":"66D53BE8A5056ECA!101","name":"root:","path":"\/drive\/root:"},"size":0}
DEBUG: Flagging object as a directory
DEBUG: Batched JSON item processing time: 1 ms, 768 μs, and 8 hnsecs
DEBUG: ------------------------------------------------------------------

So I think that there is 100% something odd with your OneDrive account that I literally cannot assist with.

At this point I would be raising a case with Microsoft to understand why this is being used and how this is being triggered:

robertschulze commented 1 month ago

In the Elements view in the browser it is indeed correctly encoded also for my OneDrives: So I guess here the browser has already applied some logic to convert from URL encoding to UTF-16. However, originally I am seeing the encoded entries in the children JSON in the Network view in the browser. E.g. here, to check, I just created a new free OneDrive (i.e. outside of my paid MS365 accounts) and created a folder Bücher: Does it look the same for you in the Network view? If not, then it must be something about how MS handles the locale of the browser, because I see no other difference.

abraunegg commented 1 month ago

@robertschulze Using one of my 'test' free Personal OneDrive accounts, from the OneDrive 'root' the folder Bücher shows the following:

Subfolders within Bücher show that the Parent 'name' without any encoding. This is the correct response. The 'path' and 'pathFromRoot' is URL encoded - as this is expected to be used with other queries:

Everything is being presented correctly for the 'name' JSON element, which the 'onedrive' client uses for all operations.

So right now, I suspect that what you are seeing is the result one of the following:

A web browser extension installed messing around with your locale
Microsoft determining that it needs to URL encode its own API JSON response based on your locale for your account.

What I have in 'My Profile' for this test account is the following:

What is yours set to? If I change my account to match yours - do I get the same experience? If you change yours to match mine does this change your experience?

In my view this is potentially a OneDrive API Bug as the 'name' element in the API responses should never be URL encoded. The OneDrive API does not specify at all that the 'name' element may contain URL encoded characters - thus this is why I think this is an API bug.

That no-one else however has reported this (only yourself and this started off with %20 characters in folder names) ... so I cannot rule out something to do with how you have configured something at your end which is a contributing item that triggers this.

robertschulze commented 1 month ago

@abraunegg I did a number of tests. Disabling all plugins in Chrome did not help, changing my country to Australia also did not help (language was already English (United States)). Now, switching to Firefox, actually I see the same as "Bücher" as well but checking the actual response (collapsing the formatted JSON / scrolling all the way down) I see that it actually is "B\u00fccher" so apparently - as opposed to Chrome - Firefox appears to perform some internal conversion. Can you maybe check on your side whether this is also the case for you? If so, could it be that the OneDrive browser GUI is using a different API than your onedrive?

abraunegg commented 1 month ago

@robertschulze As i have explained and demonstrated before - nothing is being mangled or presenting as being URL Encoded anywhere

abraunegg commented 1 month ago

@robertschulze

If so, could it be that the OneDrive browser GUI is using a different API than your onedrive?

They both use the same Microsoft Graph API

abraunegg commented 1 month ago

@ificator Do you have any idea what the contributing factor's are here? I am at a loss as to what is going on here.

Any input you have would be greatly appreciated.

robertschulze commented 1 month ago

@abraunegg You are right, i was mistaking B\u00fccher for URL encoding of Bücher, but it is just UTF. But then I think the response from the API is actually fine:

{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#drives('e7247016c7826fbc')/items('E7247016C7826FBC%21103')/children","@odata.count":8,"value":[........,{"id":"E7247016C7826FBC!2406","name":"B\u00fccher","eTag":"aRTcyNDcwMTZDNzgyNkZCQyEyNDA2LjA","cTag":"adDpFNzI0NzAxNkM3ODI2RkJDITI0MDYuNjM4NTM5ODU4NjM1NTAwMDAw","size":2816411537,"folder":{"childCount":25,"view":{"viewType":"thumbnails","sortBy":"name","sortOrder":"ascending"}},"fileSystemInfo":{"createdDateTime":"2024-04-24T18:58:17.283Z","lastModifiedDateTime":"2024-04-24T18:58:13Z"},"parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!103","name":"Dokumente","path":"/drives/e7247016c7826fbc/items/E7247016C7826FBC!103:"}},.........]}

With sync_list:

/OneDrive - Elias/*

actually nothing is synced because items within "OneDrive - Elias" are considered root items (which I think is due to the fact that they are linked even though they are linked from folder subfolder Documents within the other OneDrive):

DEBUG: Adding 8 OneDrive items for processing from the OneDrive 'root' folder
[...]
DEBUG: sync_list item to check: Bücher
DEBUG: Evaluation against 'sync_list' for this path: Bücher
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: EXCLUDED
DEBUG: Skipping item - excluded by sync_list config: Bücher

With sync_list:

/OneDrive - Elias/*
/Bücher/*

the folder Bücher is matched per se:

DEBUG: Adding 8 OneDrive items for processing from the OneDrive 'root' folder
[...]
DEBUG: Evaluation against 'sync_list' for this path: Bücher
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' entry: /Bücher/*
DEBUG: Exact path match with 'sync_list' entry
DEBUG: Evaluation against 'sync_list' result: direct match
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync

Then the list of files within the folder is retrieved:

{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#drives('e7247016c7826fbc')/items('E7247016C7826FBC%212406')/children","@odata.count":25,"value":[.............,{"id":"E7247016C7826FBC!1492","name":"20221008 - Pythagoras 5.pdf","eTag":"aRTcyNDcwMTZDNzgyNkZCQyExNDkyLjQ","cTag":"aYzpFNzI0NzAxNkM3ODI2RkJDITE0OTIuMjU3","size":186489673,"file":{"mimeType":"application/pdf","hashes":{"quickXorHash":"8sxKNhJpXoxSqeXNUYyGXFTKfbg=","sha1Hash":"B801A340E3D338DEFA1E101C0E28260184372702","sha256Hash":"1F6A3DFAE430BB380AAF71B642B7FD3899361988930F22CBFD3DA639265B25CF"}},"fileSystemInfo":{"createdDateTime":"2022-12-18T13:29:19.28Z","lastModifiedDateTime":"2022-10-08T09:03:09Z"},"parentReference":{"driveId":"e7247016c7826fbc","driveType":"personal","id":"E7247016C7826FBC!2406","name":"B\u00fccher","path":"/drives/e7247016c7826fbc/items/E7247016C7826FBC!103:/B%C3%BCcher"}},.............]}

where the name of the parentReference is fine (B\u00fccher). However, the path of the parentReference (/drives/e7247016c7826fbc/items/E7247016C7826FBC!103:/B%C3%BCcher) contains URL encoded letters.

This is why the file is not being synced

DEBUG: skip_file item to check (file name only - parent path not in database): /20221008 - Pythagoras 5.pdf
DEBUG: skip_file evaluation for: /20221008 - Pythagoras 5.pdf
DEBUG: skip_file evaluation for: /20221008 - Pythagoras 5.pdf
DEBUG: Result: false
DEBUG: CAUTION:    The JSON element transmitted by the Microsoft OneDrive API includes HTML URL encoded items, which may complicate pattern matching and potentially lead to synchronisation problems for this item.
DEBUG: WORKAROUND: An alternative solution could be to change the name of this item through the online platform: /B%C3%BCcher/20221008 - Pythagoras 5.pdf
DEBUG: See: https://github.com/OneDrive/onedrive-api-docs/issues/1765 for further details
DEBUG: sync_list item to check: B%C3%BCcher/20221008 - Pythagoras 5.pdf
DEBUG: Evaluation against 'sync_list' for this path: B%C3%BCcher/20221008 - Pythagoras 5.pdf
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: /OneDrive - Elias/*
DEBUG: Evaluation against 'sync_list' entry: /Bücher/*
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: EXCLUDED
DEBUG: Skipping item - excluded by sync_list config: B%C3%BCcher/20221008 - Pythagoras 5.pdf

(except if I would add B%C3%BCcher to the sync_list)

At the same time, your screenshot above also shows the path of the parentReference containing URL encoded letters: So I would expect the same issue to appear also in your case, is it not?

What I meant w.r.t. to Firefox: it appears to convert UTF, i.e. if you look at the RAW json response you see B\u00fccher while in the "formatted" JSON view you see Bücher. Chrome shows B\u00fccher also in the formatted JSON view. In any case, as you correctly state, no mangling of URL encodes, this is just a question of how they handle UTF.

abraunegg commented 1 month ago

@robertschulze The 'onedrive' code does not use the 'path' element

abraunegg commented 1 month ago

The whole JSON response item you need to reference is 'name' - this is the only item which is being HTML encoded in your responses. This is why:

Things are not matching
Folders are not being synced as the path, as calculated, does not equal the path name being presented.

abraunegg commented 1 month ago

@robertschulze

You are right, i was mistaking B\u00fccher for URL encoding of Bücher, but it is just UTF. But then I think the response from the API is actually fine

Not correct ... event the UTF encoding is wrong .. in ALL of my examples the 'name' JSON element contains the correct text - unmangled by either UTF encoding or HTML encoding.

The only thing I can think of is you have configured something somewhere for Microsoft to auto-translate or auto do something.

robertschulze commented 1 month ago

For my reference, could you please share the full RAW json of your "Bücher" example (from the onedrive client with --debug-https or in Firefox, selecting children on the left and on the right side scrolling all the way down to the actual json string)?

abraunegg commented 1 month ago

Details have been sent offline.

abraunegg commented 3 weeks ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

abraunegg commented 3 weeks ago

@robertschulze I am unable to communicate with you via email due to some change with your setup. .. messages cannot be delivered.

abraunegg commented 2 weeks ago

@robertschulze Your inbound email rules are still broken

abraunegg / onedrive