[alpha] Slow/unresponsive loading/navigating very large dirs

[Bfgeshka]

Current alpha freezes on start, hanging on 100% of CPU resources. Branch next is fine though.

Will provide needed info on demand.

[IgnorantGuru]

A backtrace per BUILD DEBUG instructions in README would be helpful on this, not seeing it.

[Bfgeshka]

Thread 5 (Thread 0x7fffd8bc3700 (LWP 6865)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5555558c9b98) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        __ret = -512
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x5555558c9b48, cond=0x5555558c9b70) at pthread_cond_wait.c:502
        spin = 0
        buffer = {__routine = 0x7ffff52b2500 <__condvar_cleanup_waiting>, __arg = 0x7fffd8bc2bc0, __canceltype = 575, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x5555558c9b70, mutex = 0x5555558c9b48, private = 0}
        rt = <optimized out>
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        signals = <optimized out>
        result = 0
        wseq = 0
        seq = 0
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#2  __pthread_cond_wait (cond=0x5555558c9b70, mutex=0x5555558c9b48) at pthread_cond_wait.c:655
No locals.
#3  0x00007fffdb3f528b in util_queue_thread_func () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#4  0x00007fffdb3f4fc7 in impl_thrd_routine () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#5  0x00007ffff52ab903 in start_thread (arg=<optimized out>) at pthread_create.c:465
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736829601536, -250524935137023840, 140737488347838, 140737488347839, 140736829601536, 0, 250589275258071200, 250546485170645152}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0,
              0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007ffff4fe1f2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Thread 4 (Thread 0x7fffd93c4700 (LWP 6864)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5555558c96e0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        __ret = -512
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x5555558c9690, cond=0x5555558c96b8) at pthread_cond_wait.c:502
        spin = 0
        buffer = {__routine = 0x7ffff52b2500 <__condvar_cleanup_waiting>, __arg = 0x7fffd93c3bc0, __canceltype = 575, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x5555558c96b8, mutex = 0x5555558c9690, private = 0}
        rt = <optimized out>
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        signals = <optimized out>
        result = 0
        wseq = 0
        seq = 0
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#2  __pthread_cond_wait (cond=0x5555558c96b8, mutex=0x5555558c9690) at pthread_cond_wait.c:655
No locals.
#3  0x00007fffdb3f528b in util_queue_thread_func () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#4  0x00007fffdb3f4fc7 in impl_thrd_routine () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#5  0x00007ffff52ab903 in start_thread (arg=<optimized out>) at pthread_create.c:465
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736837994240, -250524935137023840, 140737488347838, 140737488347839, 140736837994240, 0, 250590375306569888, 250546485170645152}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0,
              0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007ffff4fe1f2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Thread 3 (Thread 0x7fffd9bc5700 (LWP 6862)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5555558d0370) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        __ret = -512
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x5555558d0320, cond=0x5555558d0348) at pthread_cond_wait.c:502
        spin = 0
        buffer = {__routine = 0x7ffff52b2500 <__condvar_cleanup_waiting>, __arg = 0x7fffd9bc4bc0, __canceltype = 575, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x5555558d0348, mutex = 0x5555558d0320, private = 0}
        rt = <optimized out>
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        signals = <optimized out>
        result = 0
        wseq = 0
        seq = 0
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#2  __pthread_cond_wait (cond=0x5555558d0348, mutex=0x5555558d0320) at pthread_cond_wait.c:655
No locals.
#3  0x00007fffdb3f528b in util_queue_thread_func () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#4  0x00007fffdb3f4fc7 in impl_thrd_routine () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#5  0x00007ffff52ab903 in start_thread (arg=<optimized out>) at pthread_create.c:465
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736846386944, -250524935137023840, 140737488347518, 140737488347519, 140736846386944, 0, 250591475355068576, 250546485170645152}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0,
              0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007ffff4fe1f2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Thread 2 (Thread 0x7fffda5d2700 (LWP 6853)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5555558c87c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        __ret = -512
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x5555558c8770, cond=0x5555558c8798) at pthread_cond_wait.c:502
        spin = 0
        buffer = {__routine = 0x7ffff52b2500 <__condvar_cleanup_waiting>, __arg = 0x7fffda5d1bc0, __canceltype = 575, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x5555558c8798, mutex = 0x5555558c8770, private = 0}
        rt = <optimized out>
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        signals = <optimized out>
        result = 0
        wseq = 0
        seq = 0
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#2  __pthread_cond_wait (cond=0x5555558c8798, mutex=0x5555558c8770) at pthread_cond_wait.c:655
No locals.
#3  0x00007fffdb3f528b in util_queue_thread_func () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#4  0x00007fffdb3f4fc7 in impl_thrd_routine () from /usr/lib64/gallium-pipe/pipe_radeonsi.so
No symbol table info available.
#5  0x00007ffff52ab903 in start_thread (arg=<optimized out>) at pthread_create.c:465
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736856925952, -250524935137023840, 140737488348142, 140737488348143, 140736856925952, 0, 250583493695219872, 250546485170645152}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0,
              0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007ffff4fe1f2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Thread 1 (Thread 0x7ffff7f86980 (LWP 6717)):
#0  0x00007ffff5ffc4ad in g_list_index () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#1  0x000055555559d38e in ptk_file_list_get_path (tree_model=0x555555fef5c0, iter=<optimized out>) at ptk/ptk-file-list.c:462
        path = 0x555555ae9ac0
        l = 0x7fffbac29060
        list = 0x555555fef5c0
        __func__ = "ptk_file_list_get_path"
        _g_boolean_var_ = <optimized out>
#2  0x0000555555595e60 in ptk_file_browser_calc_sel_change (file_browser=file_browser@entry=0x555555d49a20) at ptk/ptk-file-browser.c:3826
        it = {stamp = -1329571617, user_data = 0x7fffbac29060, user_data2 = 0x7fffb8f46390, user_data3 = 0x0}
        model = 0x555555fef5c0
        path = <optimized out>
        tree_sel = 0x555555df1dc0
        file = 0x7fffb8f46390
        is_sel = <optimized out>
#3  0x00005555555fa749 in on_event_timer (main_window=0x555555a38020) at main-window.c:285
        a_browser = 0x555555d49a20
        num_pages = 12
        i = 9
        p = <optimized out>
        notebook = 0x555555a52310
#4  0x00007ffff6003579 in ?? () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#5  0x00007ffff600299a in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#6  0x00007ffff6002d70 in ?? () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#7  0x00007ffff6003092 in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#8  0x00007ffff78c8b87 in gtk_main () from /usr/lib64/libgtk-x11-2.0.so.0
No symbol table info available.
#9  0x0000555555578cbd in main (argc=<optimized out>, argv=<optimized out>) at main.c:1533
        run = <optimized out>
        err = 0x0
(gdb)

Figured out that spacefm hangs if trying to open quite big directory with thouthands of files. Since path was saved in session it hanged on startup.

[IgnorantGuru]

Thanks. I made a slight change to alpha which may impact this, but probably not. If the problem remains, I could use another backtrace similar to your one above - just want two examples to compare where it's stopping.

Also, you might try building 7e526af6 which is an earlier commit in alpha, and see if the problem existed then. The net installer can build this branch if you're not familiar.

If the problem exists in 7e526af6, might try building at cc94367f. This will help narrow down where it was introduced. I've been making some repairs to thread races recently which is probably when something was broken.

Also, approx how many files are in the dir, and are they images, video files, or other? Do you have thumbnails enabled?

[Bfgeshka]

Number of files there is approx up to 100k. They are images, but thumbnails generation is disabled, ofc. That's a lot, but spacefm from next branch was able to comprehend it.

I'll try building against these commits a bit later today

[IgnorantGuru]

Thanks. You might want to also consider if it's actually hanging or if it's merely taking a very long time to load the dir. A few backtraces at random stop points (then use 'cont' in gdb to continue execution) might help narrow that down.

[Bfgeshka]

Looks like it starts on commit b104f888b1a6e408ca27fabb6d4b5d1d30e456df for me. Tried last commit and two especially directed to, mostly looks about the same on regard of this issue.

Indeed, this hang is finite. And more - it is not only first loading that's taking too long, it also reduces overall responsiveness of UI. I've tried smalled, but still relatively big directory with ~25k of image files.

[IgnorantGuru]

Thanks for the testing and info. Another person has confirmed this behavior of delayed loading, and laggy UI (eg file selection) in large dir of 25k+ files with alpha, compared with 1.0.6 which is more immediate.

[tome-]

Maybe similar setting to "Max Pic Size to Thumbnail" would be useful - Stop calculate size of dir if it is larger than x MB or/and contains more files than y or calculate time is longer than z.