When sync_list is used, processes all OneDrive items even if not changed?

[pcolmer]

Bug Report Details

Describe the bug Until recently, I'd been using the modified version that fixed https://github.com/abraunegg/onedrive/issues/603. I have recently been hitting some problems with the onedrive client reporting errors about missing expiration data so thought I'd update to the latest version.

Now, however, whenever I run onedrive --synchronize, it seems to be processing all of the item states, e.g.:

...
Processing 206 OneDrive items to ensure consistent state due to sync_list being used
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Processing 210 OneDrive items to ensure consistent state due to sync_list being used
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Processing 219 OneDrive items to ensure consistent state due to sync_list being used
Processing 208 OneDrive items to ensure consistent state due to sync_list being used
Processing 210 OneDrive items to ensure consistent state due to sync_list being used
Processing 211 OneDrive items to ensure consistent state due to sync_list being used
Processing 223 OneDrive items to ensure consistent state due to sync_list being used
Processing 217 OneDrive items to ensure consistent state due to sync_list being used
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Processing 208 OneDrive items to ensure consistent state due to sync_list being used
Processing 210 OneDrive items to ensure consistent state due to sync_list being used
Processing 205 OneDrive items to ensure consistent state due to sync_list being used
Processing 211 OneDrive items to ensure consistent state due to sync_list being used
Processing 207 OneDrive items to ensure consistent state due to sync_list being used
Processing 207 OneDrive items to ensure consistent state due to sync_list being used
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Processing 211 OneDrive items to ensure consistent state due to sync_list being used
...

This happens even if there are no changes to be processed. This is a change from the previous version I was using and I'm curious if this is deliberate or a bug. The current approach does mean that synchronizing takes more time than it used to and I wasn't aware of any consistency issues when I was using the previous version.

Application and Operating System Details:

• OS: Linux hackbox2 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 18.04).
• Are you using a headless system (no gui) or with a gui installed? Headless
• OneDrive Account Type - personal
• DMD or LDC compiler version - DMD64 D Compiler v2.087.1
• Application configuration:

onedrive version                    = v2.3.10-7-g9d0a653
Config path                         = /home/philip.colmer/.config/onedrive
Config file found in config path    = true
Config option 'check_nosync'        = false
Config option 'sync_dir'            = /home/philip.colmer/OneDrive
Config option 'skip_dir'            =
Config option 'skip_file'           = ~*|.~*|*.tmp
Config option 'skip_dotfiles'       = false
Config option 'skip_symlinks'       = false
Config option 'monitor_interval'    = 45
Config option 'min_notify_changes'  = 5
Config option 'log_dir'             = /home/philip.colmer/.config/onedrive/log/
Config option 'sync_root_files'     = false
Selective sync configured           = true
sync_list contents:
Video

• Curl Version:

curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

To Reproduce Steps to reproduce the behavior if not causing an application crash:

1. Configure onedrive so that it has a sync list.
2. Run onedrive --synchronize - watch how it processes a lot of items.
3. Run it again - it still processes a lot of items even though nothing has changed.

Application Log Output:

Using Config Dir: /home/philip.colmer/.config/onedrive
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/philip.colmer/OneDrive
Application version: v2.3.10-7-g9d0a653
Account Type: personal
Default Drive ID: 42b71883c19fddae
Default Root ID: 42B71883C19FDDAE!353
Remaining Free Space: 605280787185
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: 42B71883C19FDDAE!353
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Skipping item - excluded by sync_list config: BBC Video
Processing 208 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 214 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Skipping item - excluded by sync_list config: BBC Video
Processing 216 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 215 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Music
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Skipping item - excluded by sync_list config: BBC Video
Processing 218 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Genealogy
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Video
Processing 205 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 206 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 220 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Genealogy
Skipping item - excluded by sync_list config: Shared photos
Processing 215 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Audio books
Skipping item - excluded by sync_list config: Shared genealogy
Skipping item - excluded by sync_list config: Genealogy
Skipping item - excluded by sync_list config: Music
Skipping item - excluded by sync_list config: E-books
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Skipping item - excluded by sync_list config: BBC Video
Processing 205 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 210 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 216 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 208 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 208 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Monty
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Skipping item - excluded by sync_list config: BBC Video
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Attachments
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Video
Processing 223 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Music
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Video
Processing 215 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 209 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Processing 216 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Processing 216 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Music
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Processing 205 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 210 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 206 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 204 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 204 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 204 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Processing 273 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Pictures
Skipping item - excluded by sync_list config: Music
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Processing 235 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: Music
Skipping item - excluded by sync_list config: Shared photos
Processing 222 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: ArrowBackup
Skipping item - excluded by sync_list config: Pictures
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Processing 213 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: J and J
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: BBC Audio
Processing 208 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: J and J
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Processing 218 OneDrive items to ensure consistent state due to sync_list being used
Skipping item - excluded by sync_list config: J and J
Skipping item - excluded by sync_list config: Shared photos
Skipping item - excluded by sync_list config: Documents
Skipping item - excluded by sync_list config: SkyDrive camera roll
Skipping item - excluded by sync_list config: BBC Audio
Processing 205 OneDrive items to ensure consistent state due to sync_list being used
...

I've only provided a partial output - it is just lots of the same over and over again.

Bug Report Checklist

• [x] Detailed description
• [x] Reproduction steps (if applicable)
• [x] Verbose Log Output

[abraunegg]

This is a change from the previous version I was using and I'm curious if this is deliberate or a bug. The current approach does mean that synchronizing takes more time than it used to and I wasn't aware of any consistency issues when I was using the previous version.

This is in response to fixing https://github.com/abraunegg/onedrive/issues/658

When using sync_list this discards 'changes' and information about directories & files as they are stored on OneDrive. If you then logically move a folder from a directory that is forced out of scope by 'sync_list' into scope by moving it into a directory that is synced by 'sync_list' the OneDrive API will only send 1 change for that move - the change that represents the top level folder which was moved - not any change record for any children.

So what happens is - the new folder gets created locally, but no content in that folder is ever downloaded. This is because OneDrive 'thinks' the folder state is the same, and that the 'logical' move is what occured on the local file system as all the files were previously present.

I spent a lot of time looking into this (5-6 local different versions all trying to force the OneDrive API to send all those changes) and in the end this really is a OneDrive API issue (not a bug) that it does not send all the changes for when a folder is moved - and I say not a bug because if not using 'sync_list' then we get this change / move correctly, handling it, and move the folder from A-> B with all the data replicating OneDrive.

So the only way around this - is that when using 'sync_list' force a full scan. When using 'sync_list' as well, it should not do a double 'full scan' which is needless & wasteful as well.

Unfortunately there is zero way around this at this point in time, without totally re-writing the client & how the actual OneDrive state is being stored, including a flag per DB entry as to if this is in scope / out of scope is probably the solution, so that when something 'bit flips' to be in-scope, it can be checked locally, and if it does not exist, go download it. This sort of approach would bring back the prior faster execution & negate having to go and check for changes.

Apologies for the impact this is causing.

This happens even if there are no changes to be processed.

This has given me an idea to enhance the above fix a little - if using 'sync_list' and 0 changes waiting, don't do the full check (pointless). Let me work on a PR and provide that for testing.

[abraunegg]

@pcolmer Please can you test the following PR as an enhancement to reduce the needless scanning:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/691/head:pr691
git checkout pr691
./configure; make clean; make;

[abraunegg]

@pcolmer Any chance in testing this PR yet?

[pcolmer]

I'm sorry I haven't yet had a chance to test this - I missed your PR comment as I was busy over the weekend. I'll find some time today.

[pcolmer]

Hi @abraunegg

I'm currently getting an error when I try to sync but please note that the error occurs on the current code as well as this PR.

I will email the full debug log to you.

Regards

Philip

[abraunegg]

@pcolmer Looked at the log file, will need to treat that as a separate issue. Marking this issue as fixed as evidenced in the debug log provided.

[abraunegg]

Closing issue as PR #691 merged into master

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.