Cinnamenu@json: "Search home folder" hangs the desktop

[Lurux]

 * Applet version/Build date: Version 4.13.8 (2021-04-10 10:10:53)
 * Cinnamon version: Cinnamon 4.8.6
 * Distribution: Linux Mint 20.1 (see also 19.3)
 * Snippet or attachment of ~/.cinnamon/glass.log: empty
 * Snippet or attachment of ~/.xsession-errors: no matching entries

Notify author of applet @fredcw

Issue Searching in the menu with the "search home folder" option enabled sometimes hangs the desktop for several seconds. On Linux Mint 19.3 this is even worse, as it seems to completely crash the desktop and I'm forced to ctrl-alt-backspace out of the session to fix it.

On the other hand, files are generally not the primary things I'm searching in the menu, and it's rather annoying when a file takes the first place in the results even if an application matches correctly.

Proposed solution

I don't know if that's the good term - but I think the file search should be done asynchronously, aka independently of the primary search mechanism.

The files would then appear only once they've been found so the menu doesn't have to wait for the file search to complete, and at the bottom of the results in order to avoid moving the items after the fact (and to make sure matching applications actually appear on top).

Anyways, thanks for your hard work !

[fredcw]

I can't reproduce the hung desktop bug in mint 19.3, are you sure there wasn't an error in .xsession-errors? - it may have begun with "enumerate:" But I'd like to figure this out because that's not a good bug.

I considered that the folder search function would probably be annoying with a mechanical hard drive which is why I included an option to disable it. There is a function to read the disk asynchronously but tbh, I'm not too familiar with asynchronous js yet and it looks a bit complicated - or maybe i'm just lazy :/

The file results shouldn't come before the app results because I gave them a lower search score so this wouldn't happen so I'm not sure what's happening there. But if the file result is also in the recently used files, then it'll get a higher score but even then the app result should come first. What is the file name, app name and search term?

[Lurux]

Hi @fredcw,

I don't have access to the 19.3 system rn, but at the very least I can tell you I checked the log last time and I'm pretty sure there was no mention of Cinnamenu in there - I would have attached it otherwise. I may have inadvertently logged out before checking, but as far as I'm aware the error would still be shown in .xsession-errors.old ? Anyways, I can virtualize the system to help troubleshoot the issue.

I understand the trouble with asynchronous programming, I guess this doesn't have to be a priority :smile:

As for the app search, I actually narrowed down the issue; it actually only happens if the files get an exact match, but not the application. The problem is that some tools like baobab or pavucontrol use a different display name in the menu, so if you search their technical name their place might be "stealed" by an exact file match.

On top of that, I always have ~/.config symlink in my home folder, so for example in this case typing "pavu" would point to pavucontrol's config files, but without the symlink it worked well.

Even if the last part is an edge case, I think this could be easily improved by also matching either the .desktop or the exec filename of the application. Should I open a separate issue for this ?

[fredcw]

The name of the .desktop file is matched as well as the keywords but these are given a lower score. I'll try increasing the score for the .desktop matches.

I originally thought about putting a time limit on the file search so that it would be aborted if it took longer than say 1/10 second, but then I thought that people might just think that it wasn't working and it would be better for people to just turn it off if it was too slow. But I think your right, asynchronous search would be the way to go.

[Lurux]

Yeah, I think the filename score should be increased indeed.

In the meantime, I actually think it would be a decent solution to let people select a maximum search duration in the applet settings. I dunno how easily you'd be able to interrupt the search process mid-way, though.

Also, I understand that the search is taking some time, but I think it shouldn't hang the rest of the desktop, should it ?

[nathan818fr]

Same here, I had to disable the option for this reason. For me, it's a freeze of ~0.5s on each letter typed. This seems to be simply caused by a "large" home folder (125G / 937k inodes).

$ time du -bh --max-depth=0 ~/
125G    /home/user/
"  0.20s user 1.10s system 99% cpu 1.305 total

$ time du --inodes --max-depth=0 ~/
937137  /home/user/
"  0.21s user 1.11s system 99% cpu 1.322 total

$ time find ~/ -name 'abcdef'
"  0.39s user 0.66s system 99% cpu 1.064 total

[Lurux]

Yeah, I didn't specify that but on my end the freezes only occur on the first search of a specific word. I guess it doesn't re-occur afterwards because the results are then saved to RAM (?)

[fredcw]

I've put in a PR which will hopefully solve the problem but tbh, I only have an SSD so I can't really test it. It may not make any difference at all.

@Lurux AFAIK, linux uses spare RAM as a HDD read cache plus the HDD itself has a read cache which would explain why you sometimes get a delay but when searching again shortly afterwards, it's quicker because the relevant blocks are already in the cache.

[fredcw]

Any improvement?

[Lurux]

I haven't had the time to test this thoroughly yesterday, and I don't have access to a Cinnamon install atm. I guess I'll test in a 19.3 live usb, see if it crashes. Hopefully it'll be ok :smile:

But I think unless you have a very large home folder the lag is tolerable, I tested without the .config symlink and it seemed to work much better.

[Lurux]

FYI, I just tested the latest version of Cinnamenu on a 19.3 live USB and was not able to crash the desktop, even with some content from my real home folder copied over.

This may not be very significant, as this means the file were probably cached into RAM as we said earlier, and I did not apply any updates onto the live USB.

I did notice some of the files I copied over took some time before showing up in the search results, too. Is that part of the solution you decided to apply ?

[fredcw]

Yes. Before, the file search was done synchronously which was obviously causing the delay but now it's done asynchronously and the normal results are displayed and when the file search is finished, the extra search results are added and the display is updated. But the problem with js is that if the applet continues with the file search, it' still going to block the UI anyway because it's not multitasking. What I've tried to do is to cancel the file search when another button is pressed thus not blocking the UI, and this is the thing which I can't test for.

I did notice some of the files I copied over took some time before showing up in the search results,

Do you mean that you type a search query and the normal results appear and then the file search results appear a second or two later? That is the expected behaviour, but not if you mean that the correct results don't show up at all in a particular search.

The two concerns are 1) you get the results you would expect to get and 2) the UI doesn't become unresponsive at any point.

[Lurux]

What I've tried to do is to cancel the file search when another button is pressed thus not blocking the UI, and this is the thing which I can't test for.

Huh. If JS isn't multitasking, wouldn't the fact of clicking a button (or outside the menu) itself be blocked by the file search in the first place ?

Do you mean that you type a search query and the normal results appear and then the file search results appear a second or two later ?

No, I meant some of the files did not appear the first (few) time I searched for them, but they did appear in subsequent searches. Are you using some kind of indexing mechanism ?

[fredcw]

Before I was using enumerate_children() to read the contents of a directory and yes, this would cause the applet to freeze until the function returned. Now I'm using enumerate_children_async(), which because it's a gio function implemented in C, it can run in parallel to the applet, i.e. it runs outside of JS. You pass a callback function to enumerate_children_async() so that when it's finished, it adds the callback to your call stack or event loop or whatever it's called, It's complicated and I'm not sure exactly how it works. Maybe the js interpreter is multitasking behind the scenes, but the js code itself is not multitasking. Js just simulates multitasking by adding functions to the event loop (call stack?) so that they get called eventually.

some of the files did not appear the first (few) time I searched for them

This is strange, I don't know what would cause that. There's no indexing, each search is like a "fresh" search in that respect and should have identical results.

[fredcw]

Btw, This is the first asynchronous and recursive function I've ever written. Check out searchDir() - lines 1094 to 1154 in applet.js

[fredcw]

@nathan818fr The home folder search doesn't search hidden directories but 0.5 second delay is still too long. Is there any change since the update?

[Lurux]

This is strange, I don't know what would cause that. There's no indexing, each search is like a "fresh" search in that respect and should have identical results.

Ok then, maybe I just didn't wait long enough for the results to appear the first few times. Or maybe enumerate_children() does use some kind of indexing under the hood; I observed searching the whole home folder usually takes about a second in the file manager, even on an SSD...

Btw, This is the first asynchronous and recursive function I've ever written. Check out searchDir() - lines 1094 to 1154 in applet.js

Haha lol, I can check out but I don't think I'm any more experimented than you in that domain :smile:

Btw, I kinda like the new "tiles" option, even if I'm probably not going to use it myself !

[Lurux]

@fredcw - Update - I've been able to try the updated applet on my main 20.1 system, and the file search still hangs a little bit. I'll be able to test this on real 19.3 (instead of a VM) fairly soon, too. :confused:

[fredcw]

How long is a little bit? Also, there shouldn't be any need to test on 19.3 now. I think I'm just going to have to put a time limit on the search until I can find a better solution. This will mean that search results may vary depending on what's in the disk cache.

[Lurux]

I haven't had the time to test this extensively, I'll be able to tell you when I get back to my main computer this w-e.

[Lurux]

@fredcw, after some testing I can still confirm the problem but also mitigate my statements about 19.3 (with solutions below).

---

On 19.3, the whole screen freezes for 30+ seconds the first time I do a search in the menu, except for the mouse pointer. This includes the desktop and any running applications. It turns out everything turns back to normal after that, I just wasn't patient enough to see this happen.

On 20.1 the same happens, but only for 2 - 5 seconds depending on the exact search.

In the end, I don't think there's anything special about the 19.3 system, it just has been installed for longer and hence accumulated more files in the ~/.config that I symlinked in my home folder.

When searching again without the symlink on my 19.3 computer (after a restart), I observed almost no lag. Re-adding the symlink and searching again then triggered the 30 second screen freeze that I mentioned earlier.

---

A possible solution to avoid this would be to ignore symlinks during the search process. Imagine if someone symlinked the root directory into their home folder ! This would be monstrous. Ignoring symlinks would also avoid showing duplicate entries when a file is linked at several places in your home folder.

I feel like the solution above could be sufficient for now, but in the long term I see two other things that could greatly help with this:

• First, use an indexing mechanism like mlocate as a backend to search files. This would avoid slowness for large home directories
• Second, as we mentioned; do the search asynchronously. Obviously, that would be the ideal setup, but I feel like this would not even really be needed if the two solutions above were implemented.

---

Any thoughts on this ? I hope this helps !

Btw, maybe this should be re-opened in the meantime ?

[fredcw]

This confirms what I had discovered, that my asynchronous recursive function simply wasn't working the way I expected. I've updated the applet and limited the search to 0.1 seconds until I can figure out how to do it properly. Obviously this means that the search results are limited to what's in the disk cache unless you have a fast SSD. Although on subsequent searches, as the disk cache is filled, you may get more results. Even with an SSD, symlinks to my .config and .cache directories takes 1 - 2 seconds to search.

So although the feature isn't much use on large directory structures, I suppose it's still of use if you a small number of files to search.

I don't think ignoring symlinks would fix it because someone may still have a large number of directories in their home folder, e.g. cloned github repositories. And although mlocate would probably work well, I don't think it's installed by default on most distros and I don't really want to burden the user with the complexity of additional dependencies.

There's no need to reopen the issue because unless I can figure out how to do the asynchronous search properly, I may just leave it as it is. i.e it's only useful if you a small number of files.

Thanks for the help.

[Lurux]

I don't think ignoring symlinks would fix it because someone may still have a large number of directories in their home folder, e.g. cloned github repositories.

If it's not too hard to implement, please do. This would at least help alleviate the problem in cases like me.

And although mlocate would probably work well, I don't think it's installed by default on most distros and I don't really want to burden the user with the complexity of additional dependencies.

Idk then, maybe you could check if it exists and fall back to the normal search if it's not.

I'm not too sure about that, but I think Cinnamon spices have a built-in way to ask the installation of additional packages - at least a few ones do this, and it's not too bad IMO.

Also, I was suggesting that because it's installed by default in LM, at least !

---

Uhm also, I just tried with the latest version from the applets menu, and the desktop still hangs on my side. Maybe the fix haven't been released yet ? :sweat:

[fredcw]

Current version is 4.13.13, the search is back on by default now. I didn't know that mint had mlocate by default (manjaro doesn't). In any case, I think it only updates daily by default so I don't think it would be suitable anyway.

Now that there's a 0.1 second time limit, it shouldn't make a difference if large directory structures are symlinked or not because it won't have time to search them anyway and if I manage to fix the UI blocking problem then it probably wouldn't be important anyway. I still can't find a way to interrupt the search process with a keypress. Js seems to prioritise callbacks and setTimeOut's over key events no matter what order they come in.

but I think Cinnamon spices have a built-in way to ask the installation of additional packages

Not that I know of. It would have to work for different package managers (apt, rpm, pacman etc).

I think that the current version with a time limit, is going to have to do for the time being.

[Lurux]

Ok I just checked - I'm still on 4.13.12, it appears the update hasn't reached the repos yet. I was afraid this didn't work ! :sweat_smile:

As for the symlinks, I still think ignoring them would be a net benefit. I we're going to have a limited amount of results, I would rather not clog them up with possibly duplicate results or random config files. Again, i don't know if it would be easy to implement - but if it is, that would be a simple improvement.

---

Anyway, I think I should stop annoying you now - I swear I don't want you to have nightmares about this every night :joy:

[fredcw]

I think I've fixed it this time (version 4.13.14) There's no time limit and it doesn't seem to block the UI. Also, with large folder structures, it may not search every folder, especially to a deep hierarchical level. I'm sure it's possible to ignore symlinks, so I'll look into it. In the meantime, the search doesn't enter hidden folders so you could rename your symlink to begin with a . if that's not otherwise a problem.

I don't want you to have nightmares.

This made me giggle. I wouldn't have known what needed fixing if you hadn't have told me so you've been a great help.

[Lurux]

4.13.13 only just reached me, and I can confirm it works really well. The results were surprisingly complete considering the timeout period compared to the time searching took before that. There's just one time were I managed to make it freeze !

Also, it seems like the file search only matches the start of filenames, aka. a file won't show up if you search the middle word. Is that correct ?

I think I've fixed it this time (version 4.13.14) There's no time limit and it doesn't seem to block the UI. Also, with large folder structures, it may not search every folder, especially to a deep hierarchical level.

As for 4.13.14, may I ask what criterion is used to decide whether a folder will be searched or not ? (just curious)

I'm sure it's possible to ignore symlinks, so I'll look into it. In the meantime, the search doesn't enter hidden folders so you could rename your symlink to begin with a . if that's not otherwise a problem.

Haha, well the whole point of using a symlink here is to avoid having to show hidden files or having to enter the path manually. No hidden symlink for ya ! :stuck_out_tongue_winking_eye:

Also, just wondering what happens if a file shows up both in the recents and in the file search ? I don't see it show up twice, so I assume you have a mechanism in place to avoid duplicate files, but if that's not the case this would be an interesting edge case to look into.

I don't want you to have nightmares.

This made me giggle. I wouldn't have known what needed fixing if you hadn't have told me so you've been a great help.

Awww, thanks :slightly_smiling_face:

[fredcw]

Yes, it's just from the start of filenames. I thought that searching for substrings may result in too many, possibly unwanted, search results but maybe I should change this, I'm not sure. I guess this also makes the search very slightly quicker.

When the home directory is searched, all its subdirectories (Desktop, Documents, Downloads etc) are put into an array. Then each of those subdirectories are then searched in turn and all subdirectories in those subdirectories are put into the array, and so on. I've limited the array size to 100 directories so that when the array is full, any more directories that are found will be ignored. So basically it'll search max ~100 directories but it'll likely search slightly more than this because each time a directory is searched, it is removed from the array thus making room for one more. When I search my /usr/share/ directory, this array grows to about 2500 folders and takes about 20-30 seconds. I figured, if I limit this array to 100 folders, that'll be very likely enough for most use cases, and people probably wouldn't want more than this anyway.

Yes, duplicate results are removed. The same file can have up to 4 results so priority is given first to Nemo fav files, then recent files, then Nemo bookmarks and then search results. ~/Desktop still comes up twice though.

[Lurux]

I can confirm 4.13.14 seems to work as intended, with file results showing up a little bit after normal results while searching. It seems like you managed to tame the asynchronous beast ? :wink:

Just curious - wouldn't it be more reliable to have an actual hard limit of 100 folders in the search instead of removing the visited folders afterwards ? It seems to me that having a large number of folders with few folders in each one could lead to a much larger amount of scanned folders overall. Also, as long as symlinks are scanned, a folder containing only a symlink to itself an no other folders could cause an endless loop if I understand correctly.

FYI, the config file for pavucontrol still show up bafore the application when I search for "pavu" in the menu. Have the search scores been updated yet ? :thinking:

[Lurux]

Update: Same for xed !

Since the file search results appear after the rest, they really should not push previously displayed content, and appear at the bottom of the list instead. Otherwise the user will press enter because the application is highlighted and end up opening a random file...

Should I open a separate issue for this ?

[fredcw]

Next update (4.14), I've fixed the pavu and xed problem. I increased the score before but not enough to make a big difference. I had to change the way it looked at the app id which is actually just the name of the .desktop file. I also made it not follow symlinks but I still not sure this is the right thing to do because maybe some people might expect it to do this. Also fixed the hard limit thing because it could actually end up searching more like 300 instead of 100 folders, and not in an easily predictable way. But I also upped it to 200 folders just in case, but that should still be quite a quick search. This will also avoid the circular link problem which I didn't think of before.

Thanks.

[Lurux]

Ok, cool. Having 200 folder hard limit with no symlink search seems pretty much perfect to me. We'll see if you get any feedback on that :slightly_smiling_face:

As for the app id, I'm not sure I understand what you changed exactly ? :thinking:

Oh, one last thing ! I know this could be another issue, but I think I can do a simple PR for that. I think "show favorites category" and "show favorites sidebar" in the settings could be merged in a single toggle between the two, as I don't see anyone enabling or disabling both at the same time. What do you think ?

---

Edit: What about actually putting the file search results below the rest, as these appear after the rest anyway ?

[fredcw]

One of the things I like about the search result being sorted is that you type the first few letters of a file name then just press return because it's at the top of the list. I think the convenience of these outweighs the disadvantages.

could be merged in a single toggle between the two

Yes, this would remove one toggle, not sure how much it actually simplifies the options though. Also, I'm guessing that at least one person will want their favs on the sidebar and in a separate category.

[Lurux]

One of the things I like about the search result being sorted is that you type the first few letters of a file name then just press return because it's at the top of the list. I think the convenience of these outweighs the disadvantages.

I mean... Yeah, that makes sense - I guess I'll have to deal with that :smile:

Yes, this would remove one toggle, not sure how much it actually simplifies the options though.

I mean, it isn't very straightforward for newcomers to adjust two separate options at different places if they want to have their favorites as a category.

Also, I'm guessing that at least one person will want their favs on the sidebar and in a separate category.

Haha, true :joy:

[mareksipos93]

I was having exactly the same problem - Searching using CInnamenu often totally frozen Cinnamon and I needed to restart it. Unchecking "Search home folder" fixed the freezings. Linux Mint 21 Cinnamon, cinnamon 5.4.12, kernel 5.15.0-56-generic

[fredcw]

@Iwitrag Do you mean that cinnamon is still freezing when using search home folder option? Do you have the latest version of cinnamenu?