Link Panel freezes desktop with folder with lots of files

[fossfreedom]

I thought I'd investigate the Link Panel option - connected it to my VirtualBox mapped Downloads folder on my host that happens to have a couple of hundred files in it.

The desktop froze for a while - when the rendering completed lots of icons overlapped with each other --> as you mentioned in #148 maybe horizontal, vertical bars in future would mean icon positions can be rendered outside the immediate panel boundary.

So then I used the panel properties and chose "Align to grid" - again the desktop froze for a while.

Looks like some threading work probably will resolve.

[spheras]

I think this is related with #124, I thought it was solved, at least in my laptop it seems. I tried with pure Threads but there are a lot of gtk stuff, and it was giving errors (i'm not too much experienced with Gtk, always learning new things)... after that I tried requiring the main thread in low priority:

        // let's sync the files found at this folder
        GLib.Idle.add_full (GLib.Priority.LOW, () => {
            this.sync_files (0, 0);
            return false;
        });

Are you experienced with Threads in vala/gtk?

[fossfreedom]

Probably somewhere along the lines of

Thread<bool> update_thread;
update_thread = new Thread<bool>.try ("syncprocess", run_sync);

private bool run_sync () {
    this.sync_files (0, 0);
    update_thread.exit(true);
}

I'll have a play over the weekend

[spheras]

yes, I tried that but it seems you can't run gtk stuff on threads, only the main thread can touch the screen (similar to android.. or js workers for instance)... Perhaps the only solution is to try to separate the Gtk code from the background/nogtk code... not sure.

[fossfreedom]

ok - I had a quick look at this. You are correct, the GTK code needs to be separated.

Gtk.Idle_add should only be used for very short running calculations. So what cannot be done is a long "read every file" ... calculate where the file should be plotted ... then create an icon and display it.

Correct me if I am wrong ... but it looks like the FolderManager.sync_files is the key part here that needs to be tweaked. sync_files seems to be called at various places so when dealing with threads, need also to worry about reentrant problems.

My initial thoughts are:

1. When sync_files is called check if "find list of files in folder" thread has been previously created.
2. If it has been previously created then set a Manager class variable to tell the "find list of files in folder" thread to start at the beginning again.
3. If it has not been previously created then create class based Thread find_list_of_files_in_folder as described in my comment above.
4. in the thread function find every file that needs to be found (take into account if need to start at the beginning again) - for each file pop the file details onto a Queue. If restarting at the beginning the empty the Queue.
5. Check if a class based Gtk.Idle thread has been previously created. If it hasn't create one (low priority).
6. If there is no more files to be found in the folder then exit(True) to quit the thread.
7. In the Gtk.Idle_add thread function, Pop one item from the queue. Calculate and add it to the display.
8. If the Gtk.Idle_add thread function no longer has any queue items then set the class var to zero, return False else return True (to continue)

@spheras I've got a few priority UB issues I need to deal with - but will try my best to have a closer look and rejig the code in between those issues.

EDIT: Probably only need to do the idle_add once per class (i.e. 5 & 8 above isnt needed)- since it is low priority there is no need ever to "return False" (to exit) there could be items to be processed at some future time and since the idle_thread is created with low_priority it isn't taking any resources.

[aljelly]

Tried to do perf when Desktop Folder froze for a while while loading a link panel with a large amount of files (this was when Desktop Folder was starting up):

40.96%     0.00%  com.github.sphe  [unknown]                         [.] 0000000000000000
            |
            ---0
               |          
               |--19.12%--0
               |          |          
               |          |--6.86%--0x1
               |          |          |          
               |          |          |--1.12%--g_object_unref
               |          |          |          
               |          |          |--0.75%--g_pointer_bit_unlock
               |          |          |          
               |          |           --0.56%--g_mutex_lock
               |          |          
               |          |--5.61%--0x2
               |          |          |          
               |          |          |--1.21%--g_mutex_lock
               |          |          |          
               |          |          |--0.76%--g_mutex_unlock
               |          |          |          
               |          |           --0.60%--g_type_check_instance_is_fundamentally_a
               |          |          
               |          |--2.73%--0x3
               |          |          |          
               |          |           --0.67%--0x180baf
               |          |          
               |           --1.62%--0
               |                     |          
               |                      --1.34%--g_object_unref
               |          
               |--6.02%--0x40
               |          |          
               |          |--4.32%--0xa00a4
               |          |          |          
               |          |          |--2.44%--g_signal_emit
               |          |          |          |          
               |          |          |           --0.52%--g_hash_table_lookup
               |          |          |          
               |          |           --0.56%--g_type_check_instance_is_fundamentally_a
               |          |          
               |           --1.67%--0x402001
               |                     |          
               |                      --0.91%--g_signal_emit
               |          
               |--4.95%--0x18
               |          |          
               |          |--3.31%--0x7c557
               |          |          |          
               |          |           --1.80%--g_signal_emit
               |          |          
               |           --1.59%--0x7c559
               |                     |          
               |                      --0.86%--g_signal_emit
               |          
               |--1.12%--0x1
               |          
               |--0.74%--0x55ac618e8eb0
               |          
                --0.60%--g_slice_free1

10.48%     0.00%  com.github.sphe  libglib-2.0.so.0.5600.3           [.] g_main_context_dispatch
            |
            ---g_main_context_dispatch
               ___lambda73__gsource_func
               __lambda73_
               |          
                --10.43%--desktop_folder_folder_manager_sync_files
                          |          
                          |--6.86%--desktop_folder_item_manager_new
                          |          |          
                          |           --6.84%--desktop_folder_item_manager_construct
                          |                     |          
                          |                      --6.81%--desktop_folder_item_view_new
                          |                                |          
                          |                                 --6.78%--desktop_folder_item_view_construct
                          |                                           |          
                          |                                            --6.64%--desktop_folder_item_view_refresh_icon
                          |                                                      |          
                          |                                                       --6.58%--desktop_folder_item_view_calculate_icon
                          |                                                                 |          
                          |                                                                  --6.58%--desktop_folder_item_manager_get_settings
                          |                                                                            |          
                          |                                                                             --6.29%--desktop_folder_folder_settings_get_item
                          |                                                                                       |          
                          |                                                                                        --4.47%--desktop_folder_item_settings_parse
                          |                                                                                                  |          
                          |                                                                                                  |--1.56%--desktop_folder_item_settings_new
                          |                                                                                                  |          |          
                          |                                                                                                  |           --1.11%--desktop_folder_item_settings_construct
                          |                                                                                                  |          
                          |                                                                                                   --0.70%--_vala_array_free
                          |          
                           --3.39%--desktop_folder_folder_settings_get_item
                                     |          
                                      --2.31%--desktop_folder_item_settings_parse
                                                |          
                                                 --0.79%--desktop_folder_item_settings_new
                                                           |          
                                                            --0.66%--desktop_folder_item_settings_construct

[aljelly]

Additionally when moving or resizing a link panel with a large amount of files on the auto arrange feature branch it hangs for like a minute:

42.93%     0.00%  com.github.sphe  libgtk-3.so.0.2200.30             [.] 0x00000000002310f0
            |
            ---0x2310f0
               0x23115b
               _desktop_folder_folder_window_on_configure_gtk_widget_configure_event
               desktop_folder_folder_window_on_configure
               |          
                --42.92%--desktop_folder_folder_manager_organize_panel_items
                          |          
                          |--42.33%--desktop_folder_folder_arrangement_organize_items
                          |          |          
                          |          |--21.50%--g_list_length
                          |          |          
                          |          |--10.64%--g_list_nth_data
                          |          |          
                          |          |--6.78%--desktop_folder_folder_settings_set_item
                          |          |          |          
                          |          |           --3.49%--desktop_folder_item_settings_parse
                          |          |                     |          
                          |          |                     |--1.24%--desktop_folder_item_settings_new
                          |          |                     |          |          
                          |          |                     |           --0.99%--desktop_folder_item_settings_construct
                          |          |                     |          
                          |          |                      --0.61%--_vala_array_free
                          |          |          
                          |           --2.47%--desktop_folder_folder_settings_get_item
                          |                     |          
                          |                      --1.76%--desktop_folder_item_settings_parse
                          |                                |          
                          |                                 --0.59%--desktop_folder_item_settings_new
                          |          
                           --0.52%--g_list_length

[spheras]

Ok, Thanks both, very good information. This is something to study detailed, with calm. It is not an excuse, just a comment: I don't know if the target of desktopfolder is to manage thousands of files on the desktop... Anyway, I agree to solve this kind of problems.

[aljelly]

I think it's also worthwhile trying to optimize the slowest parts of these functions to make any delay as little as reasonably possible. With threading we could probably also indicate that the link panel is loading (like a spinner to the right of the panel label).

[spheras]

I did a lot of work in the branch feature-threading to solve this issue. You can take a look. It is a bit unstable yet, we enter in the hell of threads. :skull:

[fossfreedom]

I'll continue to-do much more testing - but from an initial look I created a link panel to a folder that has hundreds of files and folders.

This did not freeze the desktop - I do like the little spinner animation.

All the icons piled ontop of each other - so I right clicked and chose "Reorganise Icon" - again the spinner appeared and all the icons correctly gridded themselves - again no desktop freezing.

I killed DesktopFolder and restarted - Started very fast - saw the animation on the link panel while it populated the panel - no issue with freezing.

So - works brilliantly! Superb :)

[spheras]

thanks @fossfreedom. My concern is stability, I've seen too much segment faults during the process. Before threads everything was easier, now we need to move carefully. I will try to ensure everything is ok before merging.

[spheras]

I did a merge to the master branch. Hope everything is right. (reopen in case you see something strange)