Bug: Files on OneDrive vanish after "download size mis-match" error

andreasscherbaum commented 4 months ago

Describe the bug

Kid's school is using OneDrive, and iPads. I installed "onedrive" in order to have a local backup of the data in the cloud.

Kid reported that occasionally files vanish entirely from OneDrive. Often I was able to restore a backup from last night, but ongoing work was gone. Occasionally this happened even while working on files.

Yesterday I increased the logging, and today, after the issue happened again, found the following log entry:

2024-02-28T19:03:04.248516+01:00 pascal onedrive[2656]: Downloading file Mathe 1/9a.one ... done.
2024-02-28T19:03:04.409914+01:00 pascal onedrive[2656]: Moving Mathe 1/9a.one.partial to Mathe 1/9a.one
2024-02-28T19:08:09.582656+01:00 pascal onedrive[2656]: Downloading file Mathe 1/9a.one ... done.
2024-02-28T19:08:09.747300+01:00 pascal onedrive[2656]: Moving Mathe 1/9a.one.partial to Mathe 1/9a.one

2024-02-28T19:13:10.586309+01:00 pascal onedrive[2656]: Starting a sync with OneDrive
2024-02-28T19:13:10.597344+01:00 pascal onedrive[2656]: Syncing changes and items from OneDrive ...
2024-02-28T19:13:13.063998+01:00 pascal onedrive[2656]: Downloading file Mathe 1/9a.one ... done.
2024-02-28T19:13:13.066703+01:00 pascal onedrive[2656]: Moving Mathe 1/9a.one.partial to Mathe 1/9a.one
2024-02-28T19:13:14.010623+01:00 pascal onedrive[2656]: Sync with OneDrive is complete
2024-02-28T19:18:16.453628+01:00 pascal onedrive[2656]: Downloading file Mathe 1/9a.one ... ERROR: File download size mis-match. Increase logging verbosity to determine why.
2024-02-28T19:18:16.459249+01:00 pascal onedrive[2656]: ERROR: File download hash mis-match. Increase logging verbosity to determine why.
2024-02-28T19:18:16.459346+01:00 pascal onedrive[2656]: INFO: Potentially add --disable-download-validation to work around this issue but downloaded data integrity cannot be guaranteed.
2024-02-28T19:18:16.459439+01:00 pascal onedrive[2656]: ERROR: The local file system returned an error with the following message:
2024-02-28T19:18:16.459571+01:00 pascal onedrive[2656]:   Error Message:    Mathe 1/9a.one: No such file or directory
2024-02-28T19:18:16.459635+01:00 pascal onedrive[2656]: Moving Mathe 1/9a.one.partial to Mathe 1/9a.one
2024-02-28T19:18:16.493564+01:00 pascal onedrive[2656]: Cannot move item: Mathe 1/9a.one: No such file or directory
2024-02-28T19:18:16.539598+01:00 pascal onedrive[2656]: Cannot upload file changes/creation: Mathe 1/9a.one: No such file or directory
2024-02-28T19:18:16.539680+01:00 pascal onedrive[2656]: Received inotify delete event from operating system .. attempting item deletion as requested
2024-02-28T19:18:16.539732+01:00 pascal onedrive[2656]: Deleting item from OneDrive: Mathe 1/9a.one

After the error, the entire file 9a.one is gone on the cloud.

There is no activity on the Linux system, it just occasionally pulls the files.

Operating System Details

OS is Ubuntu 23.04, onedrive is version v2.4.23-1.

Client Installation Method

From Distribution Package

OneDrive Account Type

Personal

What is your OneDrive Application Version

v2.4.23-1

What is your OneDrive Application Configuration

Configuration file successfully loaded
onedrive version                             = v2.4.23-1
Config path                                  = /home/pascal/.config/onedrive
Config file found in config path             = true
Config option 'sync_dir'                     = /home/pascal/OneDrive
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 600
Config option 'monitor_log_frequency'        = 6
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = 
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   = 
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = false
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = false
Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.8 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh/0.10.4/openssl/zlib nghttp2/1.52.0 librtmp/2.3
Release-Date: 2023-02-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=8018812k,nr_inodes=2004703,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1616956k,mode=755,inode64)
/dev/mapper/vgkubuntu-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17749)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/bare_5.snap on /snap/bare/5 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/core18_2796.snap on /snap/core18/2796 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/core18_2812.snap on /snap/core18/2812 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/core22_1033.snap on /snap/core22/1033 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/core22_1122.snap on /snap/core22/1122 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/firefox_3779.snap on /snap/firefox/3779 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/firefox_3836.snap on /snap/firefox/3836 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/gnome-42-2204_132.snap on /snap/gnome-42-2204/132 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/gnome-42-2204_141.snap on /snap/gnome-42-2204/141 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/gtk-common-themes_1535.snap on /snap/gtk-common-themes/1535 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/obsidian_22.snap on /snap/obsidian/22 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/obsidian_24.snap on /snap/obsidian/24 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/snapd_20290.snap on /snap/snapd/20290 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/var/lib/snapd/snaps/snapd_20671.snap on /snap/snapd/20671 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide,x-gvfs-hide)
/dev/mapper/vgkubuntu-root on /var/snap/firefox/common/host-hunspell type ext4 (ro,noexec,noatime,errors=remount-ro)
/dev/nvme0n1p2 on /boot type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1002 type tmpfs (rw,nosuid,nodev,relatime,size=1616952k,nr_inodes=404238,mode=700,uid=1002,gid=1002,inode64)
portal on /run/user/1002/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1002,group_id=1002)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1616956k,mode=755,inode64)
nsfs on /run/snapd/ns/firefox.mnt type nsfs (rw)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1616952k,nr_inodes=404238,mode=700,uid=1000,gid=1000,inode64)

What are all your local file system partition types

NAME                   FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                                                                                          0   100% /snap/bare/5
loop1                                                                                          0   100% /snap/core18/2796
loop2                                                                                          0   100% /snap/core18/2812
loop3                                                                                          0   100% /snap/core22/1033
loop4                                                                                          0   100% /snap/core22/1122
loop5                                                                                          0   100% /snap/firefox/3779
loop6                                                                                          0   100% /snap/firefox/3836
loop7                                                                                          0   100% /snap/gnome-42-2204/132
loop8                                                                                          0   100% /snap/gnome-42-2204/141
loop9                                                                                          0   100% /snap/gtk-common-themes/1535
loop10                                                                                         0   100% /snap/obsidian/22
loop11                                                                                         0   100% /snap/obsidian/24
loop12                                                                                         0   100% /snap/snapd/20290
loop13                                                                                         0   100% /snap/snapd/20671
nvme0n1                                                                                                 
├─nvme0n1p1            vfat        FAT32          8103-42B9                               504,1M     1% /boot/efi
├─nvme0n1p2            ext4        1.0            97c184a9-3800-449e-a074-0d460f5304ba      1,2G    18% /boot
└─nvme0n1p3            crypto_LUKS 2              5474f444-9732-41d3-b0d3-019724bde141                  
  └─nvme0n1p3_crypt    LVM2_member LVM2 001       X91m1k-jlo7-XDjb-a04E-aVQo-fWZ7-0KvmG7                
    ├─vgkubuntu-root   ext4        1.0            98f487db-0e59-4e60-ad91-1cc799b908da    583,3G    31% /var/snap/firefox/common/host-hunspell
    │                                                                                                   /
    └─vgkubuntu-swap_1 swap        1              ebee9427-c256-4502-a867-b1d0718871e2                  [SWAP]

How do you use 'onedrive'

Single user system, onedrive syncs into ~/OneDrive. The account is used on the school iPad as well, where all the changes happen. This also happens when the Linux system is idle and screen is locked. Comparing a backup shows that no other files outside of OneDrive are affected.

Steps to reproduce the behaviour

This happens occasionally, but there is no pattern. Kid reports that sometimes files vanish while working on them, sometimes over night.

I stopped the sync entirely for now, as the damage of loosing files and explaining this to teachers is larger than the benefit of having a backup.

Complete Verbose Log Output

Log included in first field

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg commented 4 months ago

@andreasscherbaum Firstly you are using an unsupported client application:

Please install the correct application for your distribution using the correct method to avoid issues and bugs that have been previously fixed.

The next point any issue or bug will not be fixed in v2.4.x as all development has moved to v2.5.x.

I strongly suggest the following:

Install the correct application version using the correct supported method. Once installed, re-validate your issue.

If your issue still persists, uninstall the 'package' and compile the application from source building 'alpha-5' as your client and again re-test / re-validate your issue. The 'alpha-5' code is ~12 months mature, and is fast approaching a 'beta|release-candidate' status.

Unfortunately I am going to close this issue due to your unsupported client version.

andreasscherbaum commented 4 months ago

I am aware that this version is what comes with Ubuntu. I also checked all release notes before writing this bug report, and none of the bugs listed have anything in resemblance with this problem.

dpkg --list | grep onedrive
ii  onedrive                                      2.4.23-1                                    amd64        folder synchronization with OneDrive

abraunegg commented 4 months ago

@andreasscherbaum As per: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md

Ubuntu Universe packages are not supported - they contain bugs and issues which have been fixed.

Please install your client the correct way to have a supported client version on Ubuntu.

abraunegg commented 4 months ago

@andreasscherbaum

I also checked all release notes before writing this bug report, and none of the bugs listed have anything in resemblance with this problem.

The closest item / issue to what you are seeing is the following: https://github.com/abraunegg/onedrive/issues/2586 - this was confirmed as an issue in v2.4.25 , but this issue was specifically around using VIM backup folders whilst editing, and no such issue has been found with v2.5.0 when tested.

Please can you:

Upgrade your client to a supported client version and re-test
If the issue still persists when using the supported client version:
- Compile from source 'alpha-5' and re-test
- https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#compilation--installation
- https://github.com/abraunegg/onedrive/discussions/2585
- https://github.com/abraunegg/onedrive/pull/2584

abraunegg commented 4 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

abraunegg / onedrive