search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
9.94k stars 860 forks source link

Bug: High CPU use on Ubuntu 22.04? #1997

Closed nrabett closed 2 years ago

nrabett commented 2 years ago

Describe the bug

htop shows that onedrive --monitor uses around two minutes of CPU time for each hour the computer is running. This appears to be much compared to other services.

Operating System Details

Linux Jammy 5.15.0-37-generic #39-Ubuntu SMP Wed Jun 1 19:16:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Client Installation Method

From 3rd Party Source (PPA, OpenSuSE Build Service etc)

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.4.18-1~np1

What is your OneDrive Application Configuration

Config option 'sync_dir'                     = /home/nrabett/OneDrive
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     =
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 150
Config option 'monitor_log_frequency'        = 5
Config option 'monitor_fullscan_frequency'   = 24
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               =
Config option 'azure_ad_endpoint'            =
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   =
Config option 'force_http_2'                 = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = false
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = false
Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.11

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1896164k,nr_inodes=474041,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=386136k,mode=755,inode64)
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19566)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/bare_5.snap on /snap/bare/5 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_1494.snap on /snap/core20/1494 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_1518.snap on /snap/core20/1518 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/firefox_1406.snap on /snap/firefox/1406 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/firefox_1377.snap on /snap/firefox/1377 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/gnome-3-38-2004_99.snap on /snap/gnome-3-38-2004/99 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/gnome-3-38-2004_106.snap on /snap/gnome-3-38-2004/106 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/gtk-common-themes_1534.snap on /snap/gtk-common-themes/1534 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snap-store_575.snap on /snap/snap-store/575 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snap-store_582.snap on /snap/snap-store/582 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_15904.snap on /snap/snapd/15904 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_16010.snap on /snap/snapd/16010 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd-desktop-integration_10.snap on /snap/snapd-desktop-integration/10 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd-desktop-integration_14.snap on /snap/snapd-desktop-integration/14 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sdc1 on /media/SEAGATEx4TB type ext4 (rw,nosuid,nodev,noexec,relatime)
/dev/sdb1 on /media/SEAGATExfjdsak4TB type ext4 (rw,nosuid,nodev,noexec,relatime,stripe=8191)
/dev/sdg1 on /media/Verbatim_1TB type ext4 (rw,nosuid,nodev,noexec,relatime)
/dev/sdf1 on /media/Seagate1TB type ext4 (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sde1 on /media/OneDrive-Backup type ext4 (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/127 type tmpfs (rw,nosuid,nodev,relatime,size=386132k,nr_inodes=96533,mode=700,uid=127,gid=133,inode64)
gvfsd-fuse on /run/user/127/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=127,group_id=133)
portal on /run/user/127/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=127,group_id=133)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=386136k,mode=755,inode64)
nsfs on /run/snapd/ns/snapd-desktop-integration.mnt type nsfs (rw)

What are all your local file system partition types

NAME  FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0 squash 4.0                                                              0   100% /snap/bare/5
loop1 squash 4.0                                                              0   100% /snap/core20/1494
loop2 squash 4.0                                                              0   100% /snap/core20/1518
loop3 squash 4.0                                                              0   100% /snap/firefox/1406
loop4 squash 4.0                                                              0   100% /snap/gnome-3-38-2004/99
loop5 squash 4.0                                                              0   100% /snap/firefox/1377
loop6 squash 4.0                                                              0   100% /snap/gnome-3-38-2004/106
loop7 squash 4.0                                                              0   100% /snap/gtk-common-themes/1534
loop8 squash 4.0                                                              0   100% /snap/snap-store/582
loop9 squash 4.0                                                              0   100% /snap/snap-store/575
loop10
      squash 4.0                                                              0   100% /snap/snapd/15904
loop11
      squash 4.0                                                              0   100% /snap/snapd/16010
loop12
      squash 4.0                                                              0   100% /snap/snapd-desktop-integration/10
loop13
      squash 4.0                                                              0   100% /snap/snapd-desktop-integration/14
sda
├─sda1
│     vfat   FAT32                 55AB-1709                             505.7M     1% /boot/efi
└─sda2
      ext4   1.0                   db1d3356-0xad-4910-97d8-cc4718cc57d4  162.3G    21% /
sdb
└─sdb1
      ext4   1.0   SEAGATEx4TB_jkl
                                   239aad4d-95d1-4161-8158-2ade98e1c2d7    1.7T    49% /media/SEAGATEx4TB_SPEIL
sdc
└─sdc1
      ext4   1.0   SEAGATEx4TB     9ea34fe9-6a0d-4eaf-8017-af50cd56dd10    1.7T    49% /media/SEAGATEx4TB
sdd
├─sdd1
│     vfat   FAT32                 4900-BC30
└─sdd2
      ext4   1.0                   92ce6cb3-3ddd-466e-87de-5738bf62e7c8
sde
└─sde1
      ext4   1.0   OneDriveBackup  13d511b5-28db-4535-84a1-065786a4219e  199.3G    51% /media/OneDrive-Backupdisk
sdf
└─sdf1
      ext4   1.0   SEAGATE_v_1TB  8f33065b-989c-41eb-82b3-7ebeb4af5dd9  461.5G    49% /media/Seagate1TB
sdg
└─sdg1
      ext4   1.0   Verbatim_1TB    489503dd-8a7b-4a0c-a7e8-a9390d07d2f0  461.4G    49% /media/Verbatim_1TB

How do you use 'onedrive'

My OneDrive contains around 12 000 files. The OneDrive folder is in my local home folder. One of my OneDrive folders is a symlink to a folder on the same disk the OS is running from. I use an rsync script to backup the contents of my OneDrive with versioning to a separate backup disk. I reduced the setting "monitor interval" to 150 seconds, and increased "monitor_fullscan_frequency'" to 24 - so apparently, the time between full scans should be the same.

Steps to reproduce the behaviour

Not applicable.

Complete Verbose Log Output

Not applicable. jk
h
hh

h
hh
NAME  FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0 squash 4.0                                                              0   100% /snap/bare/5
loop1 squash 4.0                                                              0   100% /snap/core20/1494
loop2 squash 4.0                                                              0   100% /snap/core20/1518
loop3 squash 4.0                                                              0   100% /snap/firefox/1406
loop4 squash 4.0                                                              0   100% /snap/gnome-3-38-2004/99
loop5 squash 4.0                                                              0   100% /snap/firefox/1377
loop6 squash 4.0                                                              0   100% /snap/gnome-3-38-2004/106
loop7 squash 4.0                                                              0   100% /snap/gtk-common-themes/1534
loop8 squash 4.0                                                              0   100% /snap/snap-store/582
loop9 squash 4.0                                                              0   100% /snap/snap-store/575
loop10
      squash 4.0                                                              0   100% /snap/snapd/15904
loop11
      squash 4.0                                                              0   100% /snap/snapd/16010
loop12
      squash 4.0                                                              0   100% /snap/snapd-desktop-integration/10
loop13
      squash 4.0                                                              0   100% /snap/snapd-desktop-integration/14
sda
├─sda1
│     vfat   FAT32                 55AB-1709                             505.7M     1% /boot/efi
└─sda2
      ext4   1.0                   db1d3356-04ad-4910-97d8-cc4718cc57d4  162.3G    21% /
sdb
└─sdb1
      ext4   1.0   SEAGATEx4TB_SPEI
                                   239a2d4d-95b1-4161-8158-2a8e98e1c2d7    1.7T    49% /media/SEAGATEx4TB_SPEIL
sdc
└─sdc1
      ext4   1.0   SEAGATEx4TB     9e134fe9-6a0d-4eaf-8017-af50cd56dd10    1.7T    49% /media/SEAGATEx4TB
sdd
├─sdd1
│     vfat   FAT32                 4900-BC30
└─sdd2
      ext4   1.0                   95ce6cb3-3ddd-466f-87de-5738bf62e7c8
sde
└─sde1
      ext4   1.0   OneDriveBackup  13d511b5-48db-4535-84a1-065786a4219e  199.3G    51% /media/OneDrive-Backupdisk
sdf
└─sdf1
      ext4   1.0   SEAGATE_NY_1TB  8f33065b-989c-41eb-82b3-7ebeb4af5dd9  461.5G    49% /media/Seagate1TB
sdg
└─sdg1
      ext4   1.0   Verbatim_1TB    489503dd-8a7b-4a0c-a7e8-a9390d07d2f0  461.4G    49% /media/Verbatim_1TB

Screenshots

No response

Other Log Information or Details

Output of strace -c -a: 

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 48,01    9,820039           5   1922740     19352 newfstatat
 15,57    3,185168          11    276810           pwrite64
 12,12    2,478775          14    175153           pread64
  5,12    1,047327          75     13835           fdatasync
  4,72    0,964659          20     47732           poll
  4,37    0,894361          14     60543           rt_sigaction
  3,79    0,774962          14     51827        79 read
  2,44    0,499150          66      7517           clock_nanosleep
  1,73    0,353507           9     38775           getdents64
  1,07    0,218759           7     30901           openat
  0,80    0,163204           5     30927           close
  0,09    0,018298          37       490        26 write
  0,06    0,013238          31       424           brk
  0,05    0,010526          39       266        13 futex
  0,02    0,005038           8       624           getsockname
  0,02    0,004824          45       107           ftruncate
  0,01    0,001371          24        55           lseek
  0,01    0,001261          48        26        26 connect
  0,00    0,000345          13        26           socket
  0,00    0,000254           4        52           fcntl
  0,00    0,000237           1       231           getpid
  0,00    0,000201           7        26           getsockopt
  0,00    0,000176           6        26           getpeername
  0,00    0,000054          54         1           chmod
  0,00    0,000020          20         1           restart_syscall
  0,00    0,000003           0        26        26 setsockopt
  0,00    0,000000           0         1           getrandom
------ ----------- ----------- --------- --------- ----------------
100,00   20,455757           7   2659142     19522 total

Additional context

No response

abraunegg commented 2 years ago

@nrabett

htop shows that onedrive --monitor uses around two minutes of CPU time for each hour the computer is running. This appears to be much compared to other services.

Sorry, however this is not a bug of issue. This will be the data integrity check being performed. Please read:

nrabett commented 2 years ago

Thanks for the quick reaction. So I am to assume that the 19 352 errors reported from newfstatat are not a concern?

abraunegg commented 2 years ago

@nrabett

Thanks for the quick reaction. So I am to assume that the 19 352 errors reported from newfstatat are not a concern?

Short answer: no concern from my end regarding newfstatat errors.

Long answer and explanation as to why no concern It is not a concern for me the following reasons: * You are not concerned by this because you have not provided any details or application error message * You have not provided any application log, verbose log or debug log that correlates these `newfstatat` errors to anything that the application is doing at that point in time There are 13 types of 'errors' that can be generated from that particular [system call](https://dashdash.io/2/newfstatat). Any of these, if they were generating an actual 'error' into the application, this would be error trapped and reported by the application (and in the worst case crash the application) .. to which you are not reporting or advising that the application is spitting out `ERROR: XXXXX` - so the assumption I have here is that since the application is not generating an error, because nothing is being reported - no issue. A valid question I have from that is - is that `newfstatat` call actually coming from 'onedrive' ? The code performs `stat()` calls yes (to obtain the last modified timestamp of the on-disk local item), however how this has been implemented under the hood in DMD or LDC I am not going to dig into right now - but I will make the assumption that the client is calling 'newfstatat' via LDC on Ubuntu 22.04 because of the following: ![image](https://user-images.githubusercontent.com/4956234/172953273-65a07a36-bf11-4a44-9db4-61759ef504ff.png) So .. a question for you: Are there errors being reported by the application when you are running it. This will be denoted by `ERROR: XXXXX` ? Whilst I wait for your response to that ... lets try and debug this further and try and replicate your strace concern: ### strace -p PID on my Development System running 'master' * RHEL 7.9 * x86_64 architecture * Built using DMD rather than LDC which the OBS packages use * `onedrive v2.4.18-16-g83a7907` * Configured the system to 'monitor_interval' = 60 and 'monitor_fullscan_frequency' = 12 * Let the application run through a number of sync cycles where the DB data & files on disk are checked ```text % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 50.74 8.688492 1215 7149 nanosleep 11.68 2.000547 23 84126 stat 9.91 1.696464 35 47755 write 6.94 1.189098 23 50037 2896 lstat 6.20 1.061389 26 39670 lseek 4.96 0.849148 56 14972 poll 1.91 0.326962 32 9964 4221 open 1.23 0.210559 24 8695 close 0.96 0.163645 38 4221 munmap 0.91 0.156066 27 5709 3 recvfrom 0.87 0.148625 148 1001 fsync 0.79 0.135844 23 5760 fstat 0.77 0.131244 22 5900 getdents 0.76 0.130260 31 4087 read 0.63 0.107124 25 4221 mmap 0.45 0.076695 25 2950 openat 0.23 0.038915 78 497 sendto 0.03 0.005684 32 173 26 futex 0.03 0.004437 25 177 readlink 0.01 0.002152 143 15 ftruncate 0.00 0.000451 451 1 1 connect 0.00 0.000198 198 1 restart_syscall 0.00 0.000163 27 6 sched_yield 0.00 0.000061 30 2 brk 0.00 0.000041 20 2 socket 0.00 0.000030 7 4 1 getpeername 0.00 0.000029 2 10 fcntl 0.00 0.000020 20 1 getsockname 0.00 0.000016 16 1 getsockopt 0.00 0.000000 0 2 2 access 0.00 0.000000 0 1 setsockopt 0.00 0.000000 0 1 chmod ------ ----------- ----------- --------- --------- ---------------- 100.00 17.124359 297111 7150 total ``` No `newfstatat` calls, `lstat` calls and errors are listed, but no application error is generated at all, so without debugging this further - and no application error tells me that nothing is occurring that is out of the ordinary. GLIBC = 2.17, whereas *I think* `newfstatat` was introduced with GLIBC 2.19 ### strace -p PID on my Test RaspberryPi running Raspbian using the OpenSuSE Packages * Configured the system to 'monitor_interval' = 150 and 'monitor_fullscan_frequency' = 24 * Raspbian | Debian GNU/Linux 11 (bullseye) * aarch64 architecture * Downloaded all my test data, and let it run for a few hours to ensure the DB data & files on disk are checked * Application version: `onedrive v2.4.18-1~np1` ```text % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ 46.41 2558.878777 971 2634602 51994 read 29.19 1609.306432 2286 703851 write 12.43 685.104877 773 885947 rt_sigaction 10.30 568.153071 948 599000 ppoll 0.66 36.192066 547 66045 1855 newfstatat 0.34 18.510708 1543 11995 pwrite64 0.28 15.631906 2043 7648 clock_nanosleep 0.11 6.006092 8105 741 fdatasync 0.07 3.637637 1067 3409 openat 0.05 2.624680 741 3542 close 0.04 2.222693 767 2896 getdents64 0.04 2.007957 589 3404 fstat 0.02 1.231529 1085 1135 ioctl 0.02 1.000533 378 2640 getpid 0.01 0.815564 2682 304 renameat 0.01 0.493909 1524 324 utimensat 0.01 0.412680 1228 336 fchmodat 0.01 0.386174 2970 130 130 connect 0.01 0.355941 1174 303 statfs 0.00 0.194493 817 238 pread64 0.00 0.151738 583 260 fcntl 0.00 0.113092 869 130 socket 0.00 0.098450 757 130 getpeername 0.00 0.095635 735 130 getsockname 0.00 0.085854 4088 21 mkdirat 0.00 0.084179 647 130 getsockopt 0.00 0.044946 917 49 inotify_add_watch 0.00 0.028855 1109 26 brk 0.00 0.017127 2854 6 ftruncate 0.00 0.009256 841 11 getrandom 0.00 0.002000 2000 1 unlinkat ------ ----------- ----------- --------- --------- ------------------ 100.00 5513.898851 1118 4929384 53979 total ``` There are some 'newfstatat' calls being listed, and again some errors .. but again zero application error are being reported. On your platform ~1% calls are 'error', on this Ubuntu test platform the 'error' rate is approx 2.8% .. ### strace -p PID on my Test Ubuntu 22.04 System using the OpenSuSE Packages * Configured the system to 'monitor_interval' = 150 and 'monitor_fullscan_frequency' = 24 * x86_64 architecture * Downloaded all my test data, and let it run for a few hours to ensure the DB data & files on disk are checked * Application version: `onedrive v2.4.18-1~np1` ```text % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ 50.19 22.049610 8 2681370 155946 read 30.01 13.185671 19 685816 write 9.43 4.140748 5 792431 poll 8.62 3.787102 3 1177102 rt_sigaction 0.53 0.234164 3 64629 1610 newfstatat 0.52 0.226717 31 7223 clock_nanosleep 0.25 0.111027 160 691 fdatasync 0.22 0.095679 8 10678 pwrite64 0.07 0.031082 10 3068 openat 0.04 0.015981 5 2696 getdents64 0.03 0.013513 4 3096 close 0.03 0.011968 45 261 rename 0.01 0.006191 3 1953 getsockname 0.01 0.005451 4 1113 ioctl 0.01 0.003916 20 187 pread64 0.01 0.002905 10 283 chmod 0.01 0.002764 9 282 utimensat 0.01 0.002719 10 260 statfs 0.00 0.000957 38 25 25 connect 0.00 0.000511 24 21 mkdir 0.00 0.000407 8 49 inotify_add_watch 0.00 0.000385 11 34 brk 0.00 0.000339 67 5 ftruncate 0.00 0.000269 10 25 socket 0.00 0.000253 1 245 getpid 0.00 0.000147 2 50 fcntl 0.00 0.000082 3 25 getsockopt 0.00 0.000074 2 25 getpeername 0.00 0.000045 1 25 25 setsockopt 0.00 0.000004 4 1 restart_syscall 0.00 0.000000 0 1 getrandom ------ ----------- ----------- --------- --------- ------------------ 100.00 43.930681 8 5433670 157606 total ``` Again, there are some 'newfstatat' calls being listed, and again some errors .. but again zero application error are being reported. On your platform ~1% calls are 'error', on this Ubuntu test platform the 'error' rate is approx 2.5% .. so nothing consistent or relatively close .. but there are zero application errors being generated (or file system error - or any error ....) So back to your original issue (in reality asking a question which should have been asked in Discussions) .... why do you have high CPU every hour: 1. The integrity check goes through all items in the cache database to validate the file has not changed - to ensure the integrity of the data 2. The date & time of the file as stored in the DB is compared against the date and time of the file 3. If the file has been modified locally, or is different time wise, the hash of the file is generated and compared with what is in the database - generating a hash is computationally expensive, and is only done at a last resort during the file integrity check So .. during this integrity check - are your files being reported as date & time being different (this is where a stat() call is used btw) .. and if different - then a hash is generated which is computationally expensive. The debug application logs exactly what is being done - especially if the computation of a hash is required. If the high CPU is being caused by the hash generation the debug logs will show this. So .. how do you change when this integrity check is performed? By default, the application will check for new data every 300 seconds, and perform an integrity check, post startup integrity check every 12 checks of your online data. This works out to be around once every hour using the default values. Now .. you have *changed* your online check 'monitor_interval' to 150 seconds .. OK .. however is your data online actually changing every 150 seconds? If not, please adjust this setting so that you are checking your OneDrive data online based on your actual online data rate of change. A value of 150 is too low in my opinion but you are free to configure things the way you like. A value of 300 really should be the smallest value you use which is why it is the default. Now .. how many times should this online check be done before the system performs an integrity check of your local data? If your data (online and local) is not changing that frequently, increase 'monitor_fullscan_frequency' so that you are performing the check at least once per day .. only you know how often your data is changing online and how often you want your data integrity to be checked. You may never want the data integrity to be checked post application has started up .. so use a value of '9999999999' for 'monitor_fullscan_frequency' - you will reboot / restart your laptop or application before you hit that value (to which the internal counter gets reset to 0 again) So .. to your question about being concerned about your newfstatat errors - am I concerned about your newfstatat errors .. no I am concerned. Should you be concerned about your newfstatat errors? Possibly .. the reported 'error' values are interesting - but why the error ... only doing full debug both from the 'client' and associated kernel / glibc stacks would that be gotten to the bottom of I feel. You are also running ~Crapbuntu~ Ubuntu, you are using a file system (EXT4) when there are much better alternative file systems to ensure your data integrity (such as ZFS - if you care about your data use ZFS - but you are free to choose what you use and put your data at risk), you have a number of disks mounted into your system - do those drives have physical drive issues, or cable issues, or power issues or something else ... do you have a problem - maybe, do you have a data integrity problem - maybe. Are these things I can assist with - unfortunately not. Why are there newfstatat errors - I do not know: * GLIBC Bug? * Kernel Bug? * Incorrect interpretation handling by `strace` of `newfstatat` system calls? This is most plausible. To get to the bottom of those 3 questions is beyond the scope of this application ... maybe raise this question with the 'strace' developers or maybe the kernel development team and see where that leads .. maybe you have found a kernel bug or an strace reporting bug or something else ..... but what is correct here is that there is no bug or issue or performance problem with this code or client (at least what has been demonstrated and evidenced thus far). If you *still* feel there is a performance issue with the application, please review https://github.com/abraunegg/onedrive/wiki/Generate-system-performance-data-for-performance-related-issues - take note of the following: * Uninstall the OpenSuSE Build Service package * Build the client manually following https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#dependencies-ubuntu-18x---ubuntu-22x--debian-9---debian-11---x86_64 and enabling debug symbols and install that client version with debug symbols enabled * Provide up-to-date performance data based on https://github.com/abraunegg/onedrive/wiki/Generate-system-performance-data-for-performance-related-issues#performance-data-to-collect > I use an rsync script to backup the contents of my OneDrive with versioning to a separate backup disk. If you were not using EXT4 and you were using ZFS, you could also do the following which has many additional benefits: * Snapshot your ZFS volume at appropriate intervals that you require to give you versions of data * Use `zfs send | zfs receive` to send those snapshots or data elsewhere
nrabett commented 2 years ago

Many thanks for answering my follow-up question to a rejected bug report with such detail! – Concerning the log, I didn’t include it because of privacy issues – editing away all the file names seemed like too much work, although evidently much less work than you put down answering. My logs (standard verbosity) for the last days show no errors.

What concerned me with the newfstatat messages was not only the number of errors but also the fact that newfstatat is triggered (sorry if this is an incorrect expression) almost two million times during one hour of operation, and that this apparently causes almost half of the client’s CPU use.

During a normal day, less than 40 files are usually changed or added to my OneDrive. I will follow your advice and reduce the monitor_fullscan frequency to around once a day. Reducing the monitor interval to 150 seconds seemed necessary because I didn’t like waiting for files uploaded from elsewhere to arrive at my server. If there is any way to perform an instant server request “manually” while onedrive --monitor is running in the background, I have overlooked it.

Concerning the other issues you mention, it certainly seems interesting to delve deeper into the possible causes of the apparent anomaly, but living in a country where houses are heated with electricity, I consider the CPU use to be acceptable. Thanks again!

abraunegg commented 2 years ago

@nrabett

Concerning the log, I didn’t include it because of privacy issues

Please re-read https://github.com/abraunegg/onedrive#reporting-an-issue-or-bug very carefully ................

If there is any way to perform an instant server request “manually” while onedrive --monitor is running in the background, I have overlooked it.

Please re-read https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#running-onedrive-in-monitor-mode in more detail.

abraunegg commented 2 years ago

@nrabett

Additionally:

What concerned me with the newfstatat messages was not only the number of errors but also the fact that newfstatat is triggered (sorry if this is an incorrect expression) almost two million times during one hour of operation, and that this apparently causes almost half of the client’s CPU use.

Potentially do not use Ubuntu .... Use a better distribution.

abraunegg commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.