upload-only sync fails to delete cloud-only files

neiljackson1984 commented 3 years ago

Bug Report Details

Describe the bug Starting with a consumer-grade OneDrive repository that contains several files (although I have observed the same behavior with a SharePoint Online site, and with a OneDrive for Business repository -- the behavior seems to be the same in all three cases), I run the onedrive sync client with the --synchronize and --upload-only command-line options, and specify a syncdir that is a newly-created, empty directory. Based on the description of the --upload-only behavior in the usage guide (https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#performing-a-one-way-upload-sync), I expect the sync client to delete all of the originally-existing files in the OneDrive repository, so as to make the OneDrive repository empty, to match the state of the syncdir. However, what actually happens is that the sync client does not delete any of the existing files in the OneDrive repository.

Application and Operating System Details:

Provide your OS & version (CentOS 6.x, Ubuntu 18.x etc) and the output of: uname -a I am running the docker image obtained by doing docker pull driveone/onedrive:latest
```
Linux 0204ef26a602 4.9.206-36.el6.x86_64 #1 SMP Mon Dec 9 16:05:37 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
```
Are you using a headless system (no gui) or with a gui installed? no gui
OneDrive Account Type The log output below is from a run using a consumer-grade OneDrive account, but I have also tested, and observed the same behavior, with a OneDrive for Business repository, and with a document library in a SharePoint Online site.
Did you build from source or install from a package? I am running the docker image obtained by doing "docker pull driveone/onedrive:latest"
OneDrive Application Version: Output of onedrive --version
```
onedrive v2.4.11-7-gb0771f7
```

OneDrive Application Configuration: Output of onedrive --display-config

[DEBUG] homePath: HOME environment variable set
[DEBUG] homePath: /home/onedrive
[DEBUG] configDirName: CLI override to set configDirName to: /onedrive/conf
Using 'user' Config Dir: /onedrive/conf
Using 'system' Config Dir:
[DEBUG] refreshTokenFilePath = /onedrive/conf/refresh_token
[DEBUG] deltaLinkFilePath = /onedrive/conf/delta_link
[DEBUG] databaseFilePath = /onedrive/conf/items.sqlite3
[DEBUG] databaseFilePathDryRun = /onedrive/conf/items-dryrun.sqlite3
[DEBUG] uploadStateFilePath = /onedrive/conf/resume_upload
[DEBUG] userConfigFilePath = /onedrive/conf/config
[DEBUG] syncListFilePath = /onedrive/conf/sync_list
[DEBUG] systemConfigFilePath = /config
[DEBUG] businessSharedFolderFilePath = /onedrive/conf/business_shared_folders
Configuration file successfully loaded
[DEBUG] sync_dir: No SHELL or USER environment variable configuration detected
[DEBUG] sync_dir: Getting syncDir from config value sync_dir
[DEBUG] syncDir: /onedrive/data
Using logfile dir: /onedrive/conf/
[DEBUG] Process ID: 7F61D263C260
[DEBUG] User UID:   519
[DEBUG] User GID:   522
[DEBUG] User Name:  onedrive
onedrive version                       = v2.4.11-7-gb0771f7
Config path                            = /onedrive/conf
Config file found in config path       = true
Config option 'check_nosync'           = false
Config option 'sync_dir'               = /onedrive/data
Config option 'skip_dir'               =
Config option 'skip_file'              = ~*|.~*|*.tmp
Config option 'skip_dotfiles'          = false
Config option 'skip_symlinks'          = false
Config option 'monitor_interval'       = 300
Config option 'min_notify_changes'     = 5
Config option 'log_dir'                = /onedrive/conf/
Config option 'classify_as_big_delete' = 1000
Config option 'upload_only'            = true
Config option 'no_remote_delete'       = false
Config option 'remove_source_files'    = false
Config option 'sync_root_files'        = false
Selective sync 'sync_list' configured  = false
Business Shared Folders configured     = false

Provide the version of curl you are using: Output of curl --version

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

Is your configured 'sync_dir' a local directory or a network mount point? local directory

What partition format type does your configured 'sync_dir' reside on? Output of: lsblk -f

NAME                              FSTYPE       LABEL UUID                                   MOUNTPOINT
sda                               isw_raid_mem
└─md126
├─md126p1                       ext4               e5424457-b75b-4034-99b4-6cdfeef4a2b9   /boot
└─md126p2                       LVM2_member        vJzipL-9n9G-cZrG-U6WT-eua1-K0VZ-VRbu3k
├─vg_naserver1-lv_swap (dm-1) swap               875d2e09-8dfa-44f3-88d4-f8b23aec0261   [SWAP]
├─vg_naserver1-lv_home (dm-2) ext4               6ebd8490-7e36-479c-a2a2-6bdd7207ecfe   /home
└─vg_naserver1-lv_root (dm-0) ext4               0700ff1a-a265-4029-8e4e-351eb1ae3eed   /

Explain your entire configuration setup - is the OneDrive folder shared with any other system, shared with any other platform at the same time, is the OneDrive account you use shared across multiple systems / platforms / Operating Systems and in use at the same time. The OneDrive folder is not shared or synced with any other system.

Note: Please generate a full debug log whilst reproducing the issue as per https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support and email to support@mynas.com.au

2021-May-23 23:58:56.3889547    [DEBUG] Testing network to ensure network connectivity to Microsoft OneDrive Service
2021-May-23 23:58:56.3892345    [DEBUG] Attempting to contact online service
2021-May-23 23:58:56.8340646    [DEBUG] Shutting down HTTP engine as sucessfully reached OneDrive Online Service
2021-May-23 23:58:56.8349447    Initializing the OneDrive API ...
2021-May-23 23:58:56.83505  Configuring Global Azure AD Endpoints
2021-May-23 23:58:56.835182 [DEBUG] Configured authUrl:          https://login.microsoftonline.com/common/oauth2/v2.0/authorize
2021-May-23 23:58:56.8352138    [DEBUG] Configured redirectUrl:      https://login.microsoftonline.com/common/oauth2/nativeclient
2021-May-23 23:58:56.835261 [DEBUG] Configured tokenUrl:         https://login.microsoftonline.com/common/oauth2/v2.0/token
2021-May-23 23:58:56.8352932    [DEBUG] Configured driveUrl:         https://graph.microsoft.com/v1.0/me/drive
2021-May-23 23:58:56.8353271    [DEBUG] Configured driveByIdUrl:     https://graph.microsoft.com/v1.0/drives/
2021-May-23 23:58:56.835355 [DEBUG] Configured sharedWithMeUrl:  https://graph.microsoft.com/v1.0/me/drive/sharedWithMe
2021-May-23 23:58:56.8353859    [DEBUG] Configured itemByIdUrl:      https://graph.microsoft.com/v1.0/me/drive/items/
2021-May-23 23:58:56.8354123    [DEBUG] Configured itemByPathUrl:    https://graph.microsoft.com/v1.0/me/drive/root:/
2021-May-23 23:58:56.8354379    [DEBUG] Configured siteSearchUrl:    https://graph.microsoft.com/v1.0/sites?search
2021-May-23 23:58:56.8354632    [DEBUG] Configured siteDriveUrl:     https://graph.microsoft.com/v1.0/sites/
2021-May-23 23:58:56.8354962    [DEBUG] Downgrading all HTTP operations to HTTP/1.1 by default
2021-May-23 23:58:56.8355255    [DEBUG] clientId    = d50ca740-c83f-4d1b-b616-12c519384f0c
2021-May-23 23:58:56.835552 [DEBUG] companyName = abraunegg
2021-May-23 23:58:56.835579 [DEBUG] appTitle    = OneDrive Client for Linux
2021-May-23 23:58:56.844541 Opening the item database ...
2021-May-23 23:58:56.8446965    [DEBUG] Using database file: /onedrive/conf/items.sqlite3
2021-May-23 23:58:57.5819741    [DEBUG] Configuring default new folder permissions as: 700
2021-May-23 23:58:57.582067 [DEBUG] Configuring default new file permissions as: 600
2021-May-23 23:58:57.5821411    All operations will be performed in: /onedrive/data
2021-May-23 23:58:57.5822177    [DEBUG] Configuring skip_dir ...
2021-May-23 23:58:57.5822462    [DEBUG] skip_dir: 
2021-May-23 23:58:57.5823398    [DEBUG] Configuring skip_dir_strict_match ...
2021-May-23 23:58:57.5823771    [DEBUG] skip_dir_strict_match: false
2021-May-23 23:58:57.5824036    [DEBUG] Configuring skip_dotfiles ...
2021-May-23 23:58:57.58243  [DEBUG] skip_dotfiles: false
2021-May-23 23:58:57.5824548    [DEBUG] Configuring skip_file ...
2021-May-23 23:58:57.5824828    [DEBUG] skip_file: ~*|.~*|*.tmp
2021-May-23 23:58:58.1083044    [DEBUG] Setting file permissions for: /onedrive/conf/refresh_token
2021-May-23 23:58:58.1084835    [DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive
2021-May-23 23:58:58.4796261    [DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive/root
2021-May-23 23:58:58.7918118    [DEBUG] OneDrive Account Details:      {"@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#drives\/$entity","driveType":"personal","id":"996abdde0f1aec3d","owner":{"user":{"displayName":"Neil Jackson","id":"996abdde0f1aec3d"}},"quota":{"deleted":1151898,"remaining":5368709120,"state":"normal","storagePlanInformation":{"upgradeAvailable":true},"total":5368709120,"used":0}}
2021-May-23 23:58:58.7925322    [DEBUG] OneDrive Account Root Details: {"@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#users('neil-autoscan%40outlook.com')\/drive\/root\/$entity","cTag":"adDo5OTZBQkRERTBGMUFFQzNEITExMDUuNjM3NTc0MDg1NjA0NTMwMDAw","createdBy":{"application":{"displayName":"OneDrive","id":"481710a4"},"user":{"displayName":"Neil Jackson","id":"996abdde0f1aec3d"}},"createdDateTime":"2021-05-23T23:06:03.453Z","eTag":"aOTk2QUJEREUwRjFBRUMzRCExMTA1LjA","fileSystemInfo":{"createdDateTime":"2021-05-23T23:06:03.453Z","lastModifiedDateTime":"2021-05-23T23:06:03.453Z"},"folder":{"childCount":4,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"996ABDDE0F1AEC3D!1105","lastModifiedBy":{"application":{"displayName":"OneDrive","id":"481710a4"},"user":{"displayName":"Neil Jackson","id":"996abdde0f1aec3d"}},"lastModifiedDateTime":"2021-05-23T23:16:00.453Z","name":"root","parentReference":{"driveId":"996abdde0f1aec3d","driveType":"personal"},"root":{},"size":0,"webUrl":"https:\/\/onedrive.live.com\/?cid=996abdde0f1aec3d"}
2021-May-23 23:58:58.7926457    Application version: v2.4.11-7-gb0771f7
2021-May-23 23:58:58.7926743    Account Type: personal
2021-May-23 23:58:58.7926999    Default Drive ID: 996abdde0f1aec3d
2021-May-23 23:58:58.7927235    Default Root ID: 996ABDDE0F1AEC3D!1105
2021-May-23 23:58:58.7927486    Remaining Free Space: 5368709120
2021-May-23 23:58:58.7927856    Fetching details for OneDrive Root
2021-May-23 23:58:58.7928143    [DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive/root
2021-May-23 23:58:58.9093118    [DEBUG] lastModifiedDateTime (OneDrive item): 2021-May-23 23:06:03.453Z
2021-May-23 23:58:58.9099806    OneDrive Root does not exist in the database. We need to add it.
2021-May-23 23:58:58.9100609    [DEBUG] lastModifiedDateTime (OneDrive item): 2021-May-23 23:06:03.453Z
2021-May-23 23:58:58.9101062    [DEBUG] Handing a OneDrive 'root' change
2021-May-23 23:58:58.9101382    [DEBUG] Update/Insert local database with item details
2021-May-23 23:58:58.9824436    [DEBUG] item details: Item("996abdde0f1aec3d", "996ABDDE0F1AEC3D!1105", "root", dir, "aOTk2QUJEREUwRjFBRUMzRCExMTA1LjA", "adDo5OTZBQkRERTBGMUFFQzNEITExMDUuNjM3NTc0MDg1NjA0NTMwMDAw", 2021-May-23 23:06:03.453Z, "", "", "", "", "", "", "Y")
2021-May-23 23:58:58.982585 Added OneDrive Root to the local database
2021-May-23 23:58:58.9826377    Initializing the Synchronization Engine ...
2021-May-23 23:58:58.9826786    [DEBUG] Configuring uploadOnly flag to TRUE as --upload-only passed in or configured
2021-May-23 23:58:58.9827236    [DEBUG] performSync API scan triggers
2021-May-23 23:58:58.9827514    [DEBUG] -----------------------------
2021-May-23 23:58:58.9827894    [DEBUG] fullScanRequired =                   false
2021-May-23 23:58:58.9828158    [DEBUG] syncListConfiguredFullScanOverride = false
2021-May-23 23:58:58.9828391    [DEBUG] -----------------------------
2021-May-23 23:58:58.9828618    [DEBUG] ################################################## NEW SYNC ##################################################
2021-May-23 23:58:58.9828841    Syncing changes from local path only - NOT syncing data changes from OneDrive ...
2021-May-23 23:58:58.9829268    Uploading differences of /onedrive/data
2021-May-23 23:58:58.9829539    [DEBUG] Processing DB entries for this driveId: 996abdde0f1aec3d
2021-May-23 23:58:58.9830733    [DEBUG] Attempting to calculate local filesystem path for 996abdde0f1aec3d and 996ABDDE0F1AEC3D!1105
2021-May-23 23:58:58.9831726    Processing .
2021-May-23 23:58:58.98321  [DEBUG] skip_dir evaluation for: root
2021-May-23 23:58:58.9832441    [DEBUG] No Strict Matching Enforced
2021-May-23 23:58:58.9832912    The directory has not changed
2021-May-23 23:58:58.9833589    Uploading new items of /onedrive/data
2021-May-23 23:58:58.9834898    [DEBUG] ################################################ SYNC COMPLETE ###############################################
2021-May-23 23:58:58.9835228    [DEBUG] Merge contents of WAL and SHM files into main database file

Complete Verbose Log Output see above

abraunegg commented 3 years ago

@neiljackson1984

I expect the sync client to delete all of the originally-existing files in the OneDrive repository, so as to make the OneDrive repository empty, to match the state of the syncdir. However, what actually happens is that the sync client does not delete any of the existing files in the OneDrive repository.

This is not a bug or issue. The reason for this is as follows:

The client, treats your 'OneDrive' folder online as the source of truth, not your local file system.
The client only knows what is in-sync, based on the cache, thus, will only delete things that the client knows to delete (locally or remote)

In your use case, you are expecting an 'empty' local directory to be your source of truth, but each time you are running the client, you are actually using OneDrive as your source of truth

This is why you are seeing this behavior.

There is no real easy way to just 'delete' everything like you are attempting to do. Suggest that you look at:

Using the '--local-first' option - this is why this exists to use the local state as the source of truth
- But unless everything was already previously in sync (the folders exist in the local cache db), nothing may actually get deleted remotely anyways - the client has no idea they exist
Use the online OneDrive website to delete / clear up your files.

Closing issue, as this is not a bug / issue.

neiljackson1984 commented 3 years ago

Abraunegg, thank you for the extremely rapid response. I had experimented with the --local-first option without much luck (just as you mention). Perhaps I should change this to a feature request for a "complete mirroring" feature. Is there a way to get the local cache db in sync with the cloud without having to download the files from the cloud (i.e. to put the database into the same state that it would be in after doing a --synchronize starting with an empty syncdir and downloading the entire repository from the cloud, but without actually having to perform the download.)? If there were such a command, I would be in business (perhaps this is another feature request). Thanks for the EXCELLENT software.

abraunegg commented 3 years ago

@neiljackson1984 This would really be 2 different feature requests:

Replicate OneDrive online state locally, no data transfer in either direction to get the local cache DB in sync.
Delete all files, local & remote

The issue with this is that the API does not allow you to remote top level objects via the API, if the item contains children. So, in order to do this, one would have to enumerate every single online item, then, reverse the tree submitting a delete action. This would be extremely time intensive - and not as quick as say just deleting all top level objects.

Right now however, the best solution for you, is go online, via a web browser and delete the objects you wish to delete.

abraunegg / onedrive

upload-only sync fails to delete cloud-only files #1476

Bug Report Details