sync_list inclusions skip subdirectories

tigerjack commented 3 years ago

Bug Report Details

Describe the bug sync_list file produces unexpected behaviour, skipping subdirectories.

Application and Operating System Details:

Provide your OS & version (CentOS 6.x, Ubuntu 18.x etc) and the output of: uname -a -> Linux azenb 5.10.16-arch1-1 #1 SMP PREEMPT Sat, 13 Feb 2021 20:50:18 +0000 x86_64 GNU/Linux
Are you using a headless system (no gui) or with a gui installed? -> Gui installed
OneDrive Account Type. -> Business
Did you build from source or install from a package? -> Build from source
If you installed from source, what is your DMD or LDC compiler version: dmd --version or ldmd2 --version -> DMD64 D Compiler v2.095.0
OneDrive Application Version: Output of onedrive --version -> onedrive v2.4.9-22-g42b7945
OneDrive Application Configuration: Output of onedrive --display-config -> different cases, see below
Provide the version of curl you are using: Output of curl --version -> curl 7.75.0
Is your configured 'sync_dir' a local directory or a network mount point? -> local
What partition format type does your configured 'sync_dir' reside on? Output of: lsblk -f -> ntfs

Full details

The content of the online directory under scrutiny. image

The local directory is empty.

First attempt Relevant config As suggested in some comments below this, in the sync_list file, I placed inclusions after exclusions.

Configuration file successfully loaded
config file has been updated, checking if --resync needed
onedrive version                       = v2.4.9-22-g42b7945
Config path                            = /home/simone/.config/onedrive/polimi/
Config file found in config path       = true
Config option 'check_nosync'           = true
Config option 'sync_dir'               = /mnt/internal/SharedData/Public/onedrive/polimi_wsl
Config option 'skip_dir'               = 
Config option 'skip_file'              = ~*|.~*|*.tmp|*.lock|*.fuse_hidden*|*desktop.ini|*.log|auto/*.el|GS.mkv|CC_3.avi
Config option 'skip_dotfiles'          = false
Config option 'skip_symlinks'          = false
Config option 'monitor_interval'       = 60
Config option 'min_notify_changes'     = 5
Config option 'log_dir'                = /var/log/onedrive/
Config option 'classify_as_big_delete' = 1000
Config option 'upload_only'            = false
Config option 'no_remote_delete'       = false
Config option 'remove_source_files'    = false
Config option 'sync_root_files'        = true
Selective sync 'sync_list' configured  = true
sync_list contents:
!Link*
!University/UniSannio
!University/PoliMi/Master
University/PoliMi/Master/Courses/Done/Cryptography
/*
Business Shared Folders configured     = false

However, all the directories that should be ignored get synced.

Downloading file LinkHardware/stm32/f303vc/STM32F3-Discovery_FW_V1.1.0/Libraries/CMSIS/Documentation/DSP_Lib/README.txt ... done.
[...]

Second attempt Relevant config Note that now the inclusions are placed before the exclusions in sync_list file

sync_list contents:
University/PoliMi/Master/Courses/Done/Cryptography
/*
!University/UniSannio
!University/PoliMi/Master
!Link*

but still, subdirs of Cryptography don't get downloaded, only level 1 files.

Configuration file successfully loaded
config file has been updated, checking if --resync needed
DRY-RUN Configured. Output below shows what 'would' have occurred.
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Downloading file University/PoliMi/Master/Courses/Done/Cryptography/ToRemember.txt ... done.
Downloading file University/PoliMi/Master/Courses/Done/Cryptography/Site.txt ... done.
Downloading file University/PoliMi/Master/Courses/Done/Cryptography/CourseSchedule.pdf ... done.
Uploading differences of /mnt/internal/SharedData/Public/onedrive/polimi_wsl
Uploading new items of /mnt/internal/SharedData/Public/onedrive/polimi_wsl
Skipping item - invalid name (Microsoft Naming Convention): ./Documents/desktop.ini
Skipping item - invalid name (Microsoft Naming Convention): ./WindowsDesktop/desktop.ini

Third attempt Relevant config Note the /* put at the end of the interesting directory.

sync_list contents:
University/PoliMi/Master/Courses/Done/Cryptography/*
/*
!University/UniSannio
!University/PoliMi/Master
!Link*

Same as previous attempt.

abraunegg commented 3 years ago

@tigerjack This issue is potentially being caused by you running onedrive v2.4.9-22-g42b7945 which was a PR in development of your other query.

To Resolve:

Find all 'onedrive' binaries on your system and ensure that you only have one binary installed
Rebuild your client from 'master' or from the latest release which is v2.4.10
Fix your sync_list as exclusions must come before inclusions

If you are still having an issue please provide a verbose debug log file as per the support requirements.

abraunegg commented 3 years ago

@tigerjack Regarding your log extract:

Configuration file successfully loaded
config file has been updated, checking if --resync needed
RY-RUN Configured. Output below shows what 'would' have occurred.
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Downloading file University/PoliMi/Master/Courses/Done/Cryptography/ToRemember.txt ... done.
Downloading file University/PoliMi/Master/Courses/Done/Cryptography/Site.txt ... done.
Downloading file University/PoliMi/Master/Courses/Done/Cryptography/CourseSchedule.pdf ... done.
Uploading differences of /mnt/internal/SharedData/Public/onedrive/polimi_wsl
Uploading new items of /mnt/internal/SharedData/Public/onedrive/polimi_wsl
Skipping item - invalid name (Microsoft Naming Convention): ./Documents/desktop.ini
Skipping item - invalid name (Microsoft Naming Convention): ./WindowsDesktop/desktop.ini

This log extract tells me you are running either of the following versions:

v2.4.9 with commit https://github.com/abraunegg/onedrive/commit/3ea16b619fed9b9d04c48cc80904f4910b2b4ac9
v2.4.9 with commit https://github.com/abraunegg/onedrive/commit/3ba8bb165a38df2eeacfbede3482f88932adfbf6

I know this is correct because your log extract has a log path flaw that is evident which is only in these 2 commits.

However, both of either of these version do not include the fixes for:

https://github.com/abraunegg/onedrive/pull/1269 - this is needed to support the correct path exclusions with !/ being handled correctly
https://github.com/abraunegg/onedrive/pull/1273 - this is needed to support correct /* as an inclusion, but not overriding a prior exclusion - this is why exclusion before inclusion is also very important

So in short:

Upgrade your client by rebuilding and installing the correct version
Make sure you only have one onedrive binary on your system and that it is the latest one that you built
Fix your sync_list as exclusions must come before inclusions

If you are still having an issue please provide a verbose debug log file as per the support requirements.

tigerjack commented 3 years ago

@abraunegg I updated the client and did all the steps above, but still the subdirectories don't get synced. I've tried to add also other inclusions, but I have the same behaviour I've also produced a verbose verbose log, but it's 1.1 GiB. Do I have to send it via email?

abraunegg commented 3 years ago

@tigerjack Compress the file and provide via a file download link from OneDrive via email - then I can download the file and check the debug logfile.

tigerjack commented 3 years ago

@abraunegg I think I figure out what the problem was. The /* inclusion should be put at the top of the file, then exclusions, then inclusions. I'll keep you posted.

abraunegg commented 3 years ago

@tigerjack

@abraunegg I think I figure out what the problem was. The /* inclusion should be put at the top of the file, then exclusions, then inclusions. I'll keep you posted.

This is not correct. The 'sync_list' file must be:

Exclusions
Inclusions

You also must be running a version >= 2.4.10 .. as all your details and testing is not valid from the evidence provided thus far.

Also, you have not provided an un-redacted verbose debug log as per request.

tigerjack commented 3 years ago

@abraunegg after a lot of trial and error, I came to the conclusion that the only way for it to work was to put the /* before everything else in the list, then the exclusions, then the inclusions. Putting the /* after the exclusions re-enable everything that was excluded before.

I did not send the verbose log because, as you know, it contains a lot of sensitive information (most importantly, my personal email address and all the list of files). I'll send you a log with a minimal working example in the next hours reproducing the same issue.

EDIT: of course, as I already said, I updated my onedrive version to the latest from master.

EDIT2: to support my claim, here it is a non-complete verbose example. I'll send the verbose one if you still need it. My sync_list

!Foo
!Link*
/*
Foo/Bar/Damn

The results

[DEBUG] Evaluation against 'sync_list' for: LinkTest
[DEBUG] Evaluation against 'sync_list' entry: !Foo
[DEBUG] Evaluation against 'sync_list' entry: !Link*
[DEBUG] Evaluation against 'sync_list' result: wildcard pattern matched but must be excluded
[DEBUG] Evaluation against 'sync_list' entry: /*
[DEBUG] Evaluation against 'sync_list' result: wildcard pattern match
[DEBUG] Evaluation against 'sync_list' entry: Foo/Bar/Damn
[DEBUG] Evaluation against 'sync_list' final result: included for sync
OneDrive Client requested to create remote path: ./LinkTest

The last two lines are quite relevant, even if one of the previous line says that it should be excluded.

abraunegg commented 3 years ago

@tigerjack And that verbose extract does not tell the whole story, and tells me you are potentially still running an old binary - because the debug logging output is what you have provided.

Please double and tripple check to ensure you have the correct binary installed - as right now, there is a high probability that you do not.

Also - stop being overly paranoid. I cannot access your data based on the debug log .... I will also not jepordise my professional integrity by exposing someones data .... If you feel you cannot trust because you are overly paranoid - send me an NDA or confidentiality agreement and I will sign ...

tigerjack commented 3 years ago

I've already checked more than once, but since you are not convinced, here is the result.

simone@azenb:~(master)>sudo updatedb
simone@azenb:~(master)>locate -w onedrive | grep -e "onedrive$"
/etc/logrotate.d/onedrive
/home/simone/.config/onedrive
/usr/local/bin/onedrive
/usr/local/etc/logrotate.d/onedrive
/usr/local/share/doc/onedrive
/usr/share/bash-completion/completions/onedrive
/usr/share/doc/onedrive
/usr/share/zsh/site-functions/_onedrive
simone@azenb:~(master)>/usr/local/bin/onedrive --version
onedrive v2.4.10-1-ga890951
simone@azenb:~(master)>which onedrive
/usr/local/bin/onedrive

Log output Application version: v2.4.10-1-ga890951

Privacy is not a matter of paranoia. And no offence, I don't know you neither personally nor professionally. And even if I would trust you completely, I still wouldn't have sent you the whole list of all my filenames and directories. You are not the only potential source of data leaks.

That being said, if you want, I can send the test log with the test case I made.

abraunegg commented 3 years ago

@tigerjack

Privacy is not a matter of paranoia. And no offence, I don't know you neither personally nor professionally. And even if I would trust you completely, I still wouldn't have sent you the whole list of all my filenames and directories. You are not the only potential source of data leaks.

I suggest then you have a read up on what an NDA / Confidentiality agreement provides you, with legally enforceable penalties and protections it provides you in regards to protecting your privacy with your sensitive data when interacting with 3rd parties.

As to your issue, please can you test the following PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/1293/head:pr1293
git checkout pr1293
./configure; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

When running the PR, your version should be: onedrive v2.4.10-3-gf4397e2 or greater.

I have tested this using the following 'sync_list' variations: Variation 1:

/*
!Foo
!Link*
Foo/Bar/Damn

Variation 2:

!Foo
!Link*
/*
Foo/Bar/Damn

Variation 3:

!Foo
!Link*
Foo/Bar/Damn
/*

The expectation of this configuration is:

Anything starting with 'Foo' is excluded
Anything starting with 'Link' is excluded
Foo/Bar/Damn is explicitly included
Anything else is included

In each case the result is as follows:

Using 'user' Config Dir: /home/alex/.config/onedrive-business-test/
Using 'system' Config Dir: 
Configuration file successfully loaded
config file has been updated, checking if --resync needed
DRY-RUN Configured. Output below shows what 'would' have occurred.
Deleting the saved status ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveBusinessTest
Application version: v2.4.10-3-gf4397e2
Account Type: business
Default Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1097425221014
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Updated Remaining Free Space: 1097425221014
Processing 96 OneDrive items to ensure consistent local state due to sync_list being used
Creating local directory: Foo
Creating local directory: Foo/Bar
Creating local directory: Foo/Bar/Damn
Creating local directory: Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file1.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file0.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file3.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file5.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file2.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file4.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file6.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file9.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file7.data ... done.
Downloading file Foo/Bar/Damn/0zsYrRhi5HbQCZJObbm5wcNWb6D6mHtz/file8.data ... done.
Creating local directory: Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file0.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file2.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file3.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file1.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file4.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file6.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file5.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file7.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file8.data ... done.
Downloading file Foo/Bar/Damn/1SBGWB1jBtSqnwzmQaBsaaUTM47xHGs3/file9.data ... done.
Skipping item - excluded by sync_list config: Foo/Other_Folder
Skipping item - excluded by sync_list config: LinkTest
Uploading differences of ~/OneDriveBusinessTest
Processing .
The directory has not changed
Processing Foo
The directory has not changed
Uploading new items of ~/OneDriveBusinessTest
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Updated Remaining Free Space: 1097425221014
Processing 96 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: Foo/Other_Folder
Skipping item - excluded by sync_list config: LinkTest

This process would have been made much easier had you provided a verbose debug log file as per request.

abraunegg commented 3 years ago

@tigerjack Please can you provide a response as to using this PR to resolve this issue.

github-actions[bot] commented 3 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

abraunegg / onedrive

sync_list inclusions skip subdirectories #1292

Bug Report Details

Full details