search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
10.1k stars 863 forks source link

Bug: Ignore ".folder_name" but sync "folder_name" #2796

Closed cyb3rko closed 1 month ago

cyb3rko commented 2 months ago

Describe the bug

Hi Alex, first of all thanks for this awesome piece of software, I've been using it for years now <3

I mainly sync my Android project folders between by devices. Each project folder has one .gradle and one gradle folder.
Now I want to ignore the .gradle folder and not sync it while keeping gradle in sync.

Project 1/ ├── .gradle/ ├── gradle/ └── etc/ Project 2/ ├── .gradle/ ├── gradle/ └── etc/ Project 3/ └── ...

Here's my sync_list:

!build/kotlin/*
!.kotlin/*
!venv/*
!.venv/*
!.gradle/*
!.idea/libraries/*
!__pycache__/*
!node_modules/*
!.next/*
!/Programming/Projects/Android/**/build/*
!/Programming/Projects/Android/**/.cxx/*
!/Programming/Projects/Web/**/build/*

/Programming

For .gradle it's working as expected:
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/.gradle/file-system.probe

But for some reason the sync_list above does also exclude my gradle folder as shown by one of the log entries:
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/gradle/libs.versions.toml

Am I misunderstanding something here?
Thanks for your help!

Operating System Details

Linux niko-fedo 6.10.6-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 19 14:09:30 UTC 2024 x86_64 GNU/Linux

Fedora Linux 40 (KDE Plasma)

Client Installation Method

From Distribution Package

OneDrive Account Type

Personal

What is your OneDrive Application Version

v2.4.25

What is your OneDrive Application Configuration

onedrive version                             = v2.4.25
Config path                                  = /home/niko/.config/onedrive
Config file found in config path             = true
Config option 'sync_dir'                     = /home/niko/OneDrive
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 6
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = 
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   = 
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 600
Config option 'ip_protocol_version'          = 0
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = true
sync_list contents:
!build/kotlin/*
!.kotlin/*
!venv/*
!.venv/*
!.gradle/*
!.idea/libraries/*
!__pycache__/*
!node_modules/*
!.next/*
!/Programming/Projects/Android/**/build/*
!/Programming/Projects/Android/**/.cxx/*
!/Programming/Projects/Web/**/build/*

/Programming
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = false
Config option 'webhook_enabled'              = false

What is your 'curl' version

8.6.0

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

/

What are all your local file system partition types

sdb                                                                                                                                  
├─sdb1  vfat        FAT32  BBD4-0741  579,8M  3%  /boot/efi
├─sdb2  ext4        1.0    f6337a4..  539M    38% /boot
└─sdb3  crypto_LUKS 2      2f4ba1d..                
  └─luks-2f4ba1d1.. btrfs  fedora_niko-fedo 4422bda7..  386,5G  13%  /home

How do you use 'onedrive'

/

Steps to reproduce the behaviour

Complete Verbose Log Output

Using 'user' Config Dir: /home/niko/.config/onedrive
Using 'system' Config Dir: /etc/onedrive
Configuration file successfully loaded

The use of --resync will remove your local 'onedrive' client state, thus no record will exist regarding your current 'sync status'
This has the potential to overwrite local versions of files with potentially older versions downloaded from OneDrive which can lead to data loss
If in-doubt, backup your local data first before proceeding with --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Using logfile dir: /var/log/onedrive/
Deleting the saved application sync status ...
Checking Application Version ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Using Curl defaults for all HTTP operations
Opening the item database ...
All operations will be performed in: /home/niko/OneDrive
Application version: v2.4.25
Account Type: personal
Default Drive ID: 4066fec86c8fc596
Default Root ID: 4066FEC86C8FC596!103
Remaining Free Space: 1005988714905
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Initializing the Synchronization Engine ...
Syncing changes and items from OneDrive ...
Applying changes of Path ID: 4066FEC86C8FC596!103
Updated Remaining Free Space: 1005988714905
Processing 275 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/.gradle/config.properties
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/.gradle/8.5
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/.gradle/vcs-1
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/.gradle/file-system.probe
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/.gradle/buildOutputCleanup
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/gradle/libs.versions.toml
Skipping item - excluded by sync_list config: Programming/Projects/Android/FlashDimCompose/gradle/wrapper  
...

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 2 months ago

@cyb3rko Unfortunately where the development cycle is at with v2.5.0 this will not be looked at. v2.5.0 is due for release 16th September 2024.

Please can you upgrade your client to v2.5.0 RC3 and retest your issue. Please read https://github.com/abraunegg/onedrive/discussions/2770 for details on how to build RC3.

cyb3rko commented 2 months ago

@abraunegg
I understand that, then I will just wait until 16th September and test it out on the new release if that's okay.
Otherwise I would have to upgrade my other clients to RC3 as well because of incompatibility, right?

abraunegg commented 2 months ago

@abraunegg I understand that, then I will just wait until 16th September and test it out on the new release if that's okay. Otherwise I would have to upgrade my other clients to RC3 as well because of incompatibility, right?

I would rather you find a way to test this before release, so that a fix can be made if required.

The v2.5.0 client is not backwards compatible - many things internally have changed how sync's occur, how integrity is checked - so as an abundance of caution is needed. Please read the client architecture documentation: https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-release-candidate-3/docs/client-architecture.md

Please find a way to test this issue with RC3 before the 16th.

cyb3rko commented 2 months ago

Tested with the newest version 'onedrive v2.5.0-rc3-58-g0d3d416'.
The issue persists:

Skipping file - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/.gradle/config.properties
Skipping path - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/.gradle/8.5
Skipping path - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/.gradle/vcs-1
Skipping file - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/.gradle/file-system.probe
Skipping path - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/.gradle/buildOutputCleanup
Skipping file - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/gradle/libs.versions.toml
Skipping path - excluded by sync_list config: ./Programming/Projects/Android/FlashDimCompose/gradle/wrapper
abraunegg commented 2 months ago

Thanks - will look at this with priority this week.

abraunegg commented 1 month ago

@cyb3rko

Regarding this issue, I believe I have this sorted via PR https://github.com/abraunegg/onedrive/pull/2797

From this PR, this is the logging output:

Skipping path - excluded by sync_list config: Issue_2796/Programming
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Web
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/Web
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/App1
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/App2
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/App2/gradle
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/Web/.gradle
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/Web/gradle
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/App2/.gradle
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/App1/gradle
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/App1/.gradle
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/Web/build
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/Web/.cxx
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/App1/.cxx
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/App1/build
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/App2/.cxx
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Android/App2/build
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/sub_folder_a
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/subfolder_b
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/Web/must_include
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/App1/must_include
Including path - included by sync_list config: Issue_2796/Programming/Projects/Android/App2/must_include
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1
Including file - included by sync_list config: Issue_2796/Programming/Projects/Android/App2/gradle/libs.versions.toml
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Android/App2/gradle
Including file - included by sync_list config: Issue_2796/Programming/Projects/Android/Web/gradle/libs.versions.toml
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Android/Web/gradle
Including file - included by sync_list config: Issue_2796/Programming/Projects/Android/App1/gradle/libs.versions.toml
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Android/App1/gradle
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/sub_folder_a/build
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/subfolder_b/build
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/sub_folder_a/.cxx
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/subfolder_b/.cxx
Including file - included by sync_list config: Issue_2796/Programming/Projects/Android/Web/must_include/data_to_include.data
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Android/Web/must_include
Including file - included by sync_list config: Issue_2796/Programming/Projects/Android/App1/must_include/data_to_include.data
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Android/App1/must_include
Including file - included by sync_list config: Issue_2796/Programming/Projects/Android/App2/must_include/data_to_include.data
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Android/App2/must_include
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2
Including file - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/sub_folder_a/.cxx/build.junk
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Web/folder_a/sub_folder_a/.cxx
Including file - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_a/subfolder_b/.cxx/build.junk
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Web/folder_a/subfolder_b/.cxx
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/.cxx
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/tier_3
Including file - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/.cxx/build.junk
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/.cxx
Including path - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/tier_3/must_include
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/tier_3/build
Including file - included by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/tier_3/must_include/data_to_include.data
Parental Path structure needs to be created to support included file: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/tier_3/must_include
Skipping path - excluded by sync_list config: Issue_2796/Programming/Projects/Web/folder_b/tier_1/tier_2/tier_3/build/include_under_build

The application config was:

Selective sync 'sync_list' configured        = true
sync_list config option 'sync_root_files'    = false
sync_list contents:
# EXCLUDE
!.gradle/*
!/Issue_2796/Programming/Projects/Android/**/build/*
!/Issue_2796/Programming/Projects/Android/**/.cxx/*
!/Issue_2796/Programming/Projects/Web/**/build/*

# INCLUDE
/Issue_2796/Programming/*

# INCLUDE ANY PDF FILE ANYWHERE
*.pdf

# INCLUDE ALL TEXT FILES IN SPECIFID DIRECTORY
/new_local_dir/*.txt

Please can you URGENTLY test this.

To test this PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2797/head:pr2797
git checkout pr2797

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

To install the PR, you will need to perform sudo make install to install the PR version.

When running the PR, your version should be: onedrive v2.5.0-rc3-62-gc82a285 or greater

cyb3rko commented 1 month ago

Thanks for the fix. I will test it out when I come home from work.

cyb3rko commented 1 month ago

@abraunegg I've just tested it and it works as expected, thank you a lot! 🚀

abraunegg commented 1 month ago

@cyb3rko Thanks for the confirmation of the issue being resolved.

I need to run that urgently through all my existing 'sync_list' test cases before merging into v2.5.0 RC3.

abraunegg commented 1 month ago

PR merged into RC3

abraunegg commented 1 month ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.