search

lxqt / libfm-qt

Core library of PCManFM-Qt (Qt binding for libfm)
https://lxqt-project.org
GNU Lesser General Public License v2.1
73 stars 52 forks source link

pcmanfm-qt is very slow to display file listings on remote filesystems #612

Closed jonesmz closed 3 years ago

jonesmz commented 3 years ago
Expected Behavior

When displaying a directory with a large number of files, which can be anywhere from a few dozen, to a few dozen thousand, pcmanfm-qt should start displaying files, even if the list is incomplete, within a second.

Current Behavior

When navigating to a folder that is hosted on another system, such as a mounted sshfs or smb share, pcmanfm-qt takes a noticeably long amount of time to display any contents of the folder, which is worse the larger the number of files in the folder.

One folder with 50 files in it takes 5 seconds to load. Another with 10,000 takes approximately 3 minutes.

Notably, however, navigating to a local folder with a similar number of files takes substantially less time. E.g. 50 files displays instantly, 10,000 displays in 5 seconds or so.

Possible Solution

It appears to be the case that pcmanfm-qt is conducting synchronous file listing operations where each operation needs to complete before any display is allowed.

Given that I can use "ls" on the commandline to get the full directory contents immediately, it appears that pcmanfm-qt is also probing the files for additional metadata of some sort, which accounts for the longer delay when dealing with remote systems.

One solution might be to display the list of files immediately, but then run the additional metadata queries asyncronously, and "fixup" what's displayed with the information as it becomes available.

And the other solution might be to batch file queries so that only the first dozen files or so need to be known before anything is displayed, and then as other files' information comes available, add them to the directory listing.

I thought that the problem might be found in folderview, foldermodelitem, folderitemdelegate, and so on, but a quick reading of those files doesn't turn up anything obvious to me (Not an expert on Qt)

Steps to Reproduce (for bugs)
  1. Create local folder with 10,000 files, with any arbitrary names and contents
  2. Navigate to that folder in pcmanfm-qt
  3. Observe that displaying the contents of the folder completes within a handful of seconds.
  4. Setup a remote filesystem mount via either sshfs or smb to another system
  5. Create 10,000 files with any arbitrary names and contents on that remote filesystem mount
  6. Navigate to that folder in pcmanfm-qt
  7. Observe that displaying the contents of the folder does not complete within a handful of seconds

For the case of a smb mount, the equivalent file display behavior on Windows 10 is nearly instant for the first few hundred files, and the rest of the files are loaded in asynchronously

Context

Normal file browsing is effected. For example, if I need to navigate to a folder that is a child of a folder with many files, i have to wait until the list of files in the parent folder is fetched before i can double click the folder that I really want.

System Information
emerge --info lxqt-meta
Portage 3.0.9 (python 3.9.0-final-0, default/linux/amd64/17.1, gcc-9.3.0, glibc-2.32-r3, 5.4.66-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-5.4.66-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_E3-1240_V2_@_3.40GHz-with-glibc2.32
KiB Mem:    32850180 total,   1410064 free
KiB Swap:    8388604 total,   8379388 free
Timestamp of repository gentoo: Thu, 31 Dec 2020 19:35:13 +0000
Head commit of repository gentoo: cc0a1b8dcca8a70c623a163234d3cb689f109d38

Head commit of repository lto-overlay: 7b7e6e82064ecc7032647a71ade2bdf90f5e8e0a

Head commit of repository mv: f3e86fc7dc67e14f18f563fcb4025c49ca7e23cc

sh bash 5.0_p18
ld GNU ld (Gentoo 2.34 p6) 2.34.0
distcc 3.3.3 x86_64-pc-linux-gnu [disabled]
ccache version 4.1 [disabled]
app-shells/bash:          5.0_p18::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18::gentoo, 3.7.9::gentoo, 3.8.6::gentoo, 3.9.0::gentoo
dev-util/ccache:          4.1::gentoo
dev-util/cmake:           3.17.4-r1::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/sandbox:         2.20::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.16.2-r1::gentoo
sys-devel/binutils:       2.34-r2::gentoo
sys-devel/gcc:            9.3.0-r2::gentoo
sys-devel/gcc-config:     2.3.2-r1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.32-r3::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: git://anongit.gentoo.org/repo/sync/gentoo.git
    priority: -1000

lto-overlay
    location: /var/db/repos/lto-overlay
    sync-type: git
    sync-uri: https://github.com/InBetweenNames/gentooLTO.git
    masters: gentoo mv

mv
    location: /var/db/repos/mv
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/user/mv.git
    masters: gentoo

Installed sets: @archive, @creative, @desktop-applications, @lxqt, @pc-base-system, @portage, @vcs
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin -march=x86-64 -mtune=generic -O2 -pipe -ffat-lto-objects"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/grs/systems.conf /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin -march=x86-64 -mtune=generic -O2 -pipe -ffat-lto-objects"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS=" --jobs --keep-going --newuse --changed-deps --deep --tree --backtrack=3000 --complete-graph --with-bdeps=y --binpkg-respect-use=y --binpkg-changed-deps=y --changed-slot=y --usepkg=y"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg clean-logs compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms split-elog split-log splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j10"
PKGDIR="/var/cache/binpkgs"
PORTAGE_COMPRESS="xz"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi alsa amd64 branding bzip2 cairo cdda cdr clang crypt dbus dri dts dvd dvdr egl emboss encode exif flac gif glamor gles2 gnome-keyring gpm gstreamer gtk gui hardened iconv icu ipv6 jpeg lcms libglvnd libnotify libtirpc mad mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf pie png policykit ppds pulseaudio qt5 readline samba sdl seccomp sound spell split-usr ssl ssp startup-notification svg systemd theora threads tiff truetype udev udisks unicode upower usb vaapi vorbis wayland widevine x264 xattr xcb xinerama xml xtpax xv xvid zeroconf zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx sse sse2 mmxext" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="coreboot efi-64 emu qemu pc" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-US" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_8" PYTHON_TARGETS="python2_7 python3_8 python3_9" QEMU_SOFTMMU_TARGETS="arm aarch64 x86_64" QEMU_USER_TARGETS="arm aarch64 x86_64" RUBY_TARGETS="ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="vesa modesetting radeon radeonsi amdgpu" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

lxqt-base/lxqt-meta-0.15.0::gentoo was built with the following:
USE="display-manager filemanager lximage nls policykit sddm trash -about -admin -archiver -minimal -powermanagement -processviewer -screenshot -ssh-askpass -sudo -terminal" ABI_X86="(64)"
CFLAGS="-O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin -march=x86-64 -mtune=generic -O2 -pipe -ffat-lto-objects -Wl,-O1 -Wl,--as-needed"
CXXFLAGS="-O2 -pipe -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin -march=x86-64 -mtune=generic -O2 -pipe -ffat-lto-objects -Wl,-O1 -Wl,--as-needed"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg clean-logs compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news nostrip parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms split-elog split-log strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin -march=x86-64 -mtune=generic -O2 -pipe -ffat-lto-objects"
tsujan commented 3 years ago

→ https://github.com/lxqt/pcmanfm-qt/issues/462#issuecomment-458921464

And please avoid long-winded comments!

jonesmz commented 3 years ago

How was my comment long winded?

tsujan commented 3 years ago

How was my comment long winded?

See https://github.com/lxqt/pcmanfm-qt/issues/462, which tells about the same thing.

jonesmz commented 3 years ago

Yes, I understand that the referenced bug is the original report for this problem, I just don't understand your request to not leave long winded comments. Could you elaborate on why you requested I not leave long winded comments?