search

linuxmint / nemo

File browser for Cinnamon
GNU General Public License v2.0
1.2k stars 298 forks source link

nemo significantly slower opening folder content #1907

Open howdev opened 6 years ago

howdev commented 6 years ago 
 * Nemo version (3.8.5)
 * Is issue with filemanager nemo
 * Distribution - Mint 19
 * Graphics hardware *and* driver (unrelated)
 *  64 bit

Issue This issue has been reported prior to Mint 18. Nemo open folders very slow. slower than Nautilus. much slower than Thunar. We expected the issue to be resolved in Mint 19, but still hasn't

Steps to reproduce Open nemo filemanager open some folder with contents. Especially can be test with /usr/bin directory. While Thunar opens instantly, there is a significant delay with nemo. to browse any folder with low item count still can see blank page then folders showing.

Expected behaviour instant show of folder contents

Other information

petermoo commented 3 years ago

I now have a clean install of LM 20 Cinnamon with all updates applied. The default 4.8.6. List view, not icon view. No sign of any big slowdown with Nemo. Differences from last post. New LM. Newer SSD. Icons and media plugin never used. Extra GBs of memory. File system cache is 1.6 GB and there are over 2 GB of memory unused. No paging.

I have not tested Nemo from a fresh boot before file cache built. I usually close Nemo when not in use just to reduce the number of applications listed across the bottom of the screen. I run trim and some other cleanups every few days, keeping the file system clean and the SSD ready for writes. I also shut down at night and boot each day.

dodona2 commented 3 years ago

I can't agree with the actual thunar comparison.

It seems so, that the problem has been solved. Thanks a lot!

nick-s-b commented 3 years ago

@flansuse I've tried all of the major DEs and all of the FMs. To me, KDE is unbearable and feels like a Windows knockoff. I just can't get used to it after using it for a full week. Its FM, while good, is lacking many things I like about nemo. After trying all these DEs and FMs for at least a week, I can tell you with certainty that none of them are perfect or bug-free. They all have issues and when you pick one over the other, you're just making compromises.

I currently use nemo because even with all of its bugs, it's still better than alternatives. And I also use Nautilus for searching. Searching in nemo is completely broken and unusable. It used to drive me nuts so I just switched to Nautilus for when I need to find something. I also use ranger in terminal but a lack of a decent preview means it's only not that useful to me. Anyway, experiment and see what works for you.

Jeremy7701 commented 3 years ago

I use locate + grep for any serious searching by name - its probably the only way to get a really fast name search - prebuild an index (in background) at the beginning of the day. Nemo searching in a reasonably sized folder is perfectly adequate.

snoopdouglas commented 2 years ago

In Mint 20.3, I'm finding that Nemo is generally fast at everything but opening folders. This is something I've tested with all add-ons disabled.

When navigating, there's a 0.3-0.5s pause before anything displays whenever I make a navigation.

jondo commented 2 years ago

I'm on Mint 20.3 Cinnamon with its Nemo 5.2.4. I have got a folder of 120,000 small images (up to 570 kB, but most < 40 kB), and I am suffering heavily - even with deactivated thumbnails. I see this on ext4 and btrfs, both on SSD. Also, thumbnails would be quite useful here.

My project partner is using Windows 10 with similar (or even older) hardware and has no problems at all with this folder (even with the thumbnails), so he doesn't want to change the directory structure :-/

Jeremy7701 commented 2 years ago

If you create your thumbnails over a period of time, they will appear relatively quickly. If you ask nemo to call the thumbnailer program 120,000 times it will take a little time.

I wonder why the sub-folder feature was invented?

mindinsomnia commented 2 years ago

I wonder why the sub-folder feature was invented?

Just saying, other file browsers, such as the file browser of Windows 10, have absolutely no issue loading a list of 120,000 files or more.

I'm on Windows 10 right now, and if I go to C:\ and search '*', results begin to instantly, and in a couple of seconds exceed 200,000 files. I can scroll through all of them, change between views, from tiled list to max size thumbnails, with what feels like almost no lag at all. Scrolling has an instant response time, and thumbnails for images within the scrolling view appear in less than 1 or 2 seconds.

.. About 30 seconds has passed now, and it's showing roughly 600,000 files and again, still no noticeably troubling lag to worry about.. I can drag the scroll bar from the top to the bottom and back again without an issue. Still no issues.

If I did this in Nemo, Nemo would have probably frozen and crashed by now, assuming it managed to get up to 600,000 files in the first place.

The issue isn't so much that it takes time to generate file thumbnails, the issue is that Nemo can't seem to handle processing the work of displaying a large number of files, in a way that doesn't result in extreme lag, unresponsiveness and eventual crashes.

Just looking at the Windows file manager again, and it's now up to about 1,200,000 files, and I can scroll through them with no issues. As I scroll to files such as image files, thumbnails are being generated and loaded for what's on screen very quickly.

So something Nemo is doing must be very inefficient if other file managers can handle millions of files, while Nemo (from my own experiences) can barely handle 10,000 before lag and crashes start to become an issue.

We should investigate what is causing this lag.

Here are just a few questions to put out there to anyone familiar with the codebase of Nemo for possible causes:

(Disclaimer: I support the Linux Mint project, I donate to it, and I love Linux Mint, and Nemo is my preferred file manager for Linux, because I feel the UI is probably the nicest looking. But Nemo's performance is definitely still an issue in Mint, even if it has improved a bit in recent years.)

petermoo commented 2 years ago

Are your speed tests immediately after a boot when the file cache is empty or later when there is stuff in the file cache?

I find Linux slow for file searches the first time then fast after the file cache is loaded. On some older versions of Windows, the fast initial speed appears to be a background read ahead for directory and file information. Just a theory.

The test would be to run the same 120,000 file process on Windows and Linux after a clean boot then repeat the test a couple of times after the file cache is full. I do not have dual boot for a good comparison.

On my machine, there is about 14 GB of RAM for file cache and that only just fills up with a full disk file search by name of about 500,000 files. The results will vary if the file cache cannot hold all the file/directory info.

Jeremy7701 commented 2 years ago

Thumbnails are stored in ~/.cache/thumbnails/ as PNG files. I've noticed that F5 causes an apparent rebuild of thumbnails. Initial build of thumbnails for PDF, video, EPUB files seem to take the longest.

ranolfi commented 2 years ago

Hmm - PNG, huh? I never paid attention to that.

I see an opportunity of massive performance improvement there. Just tested with a few samples from my normal and large thumbnail folders and was able to get 75% - 80% reductions in file size converting them to JPG with 90% quality. In the worst scenario, I got a 12.5% increase for some thumbnails at the very low end of filesize (< 3KB).

While this may not be the core factor to the issue being discussed here, I wonder how much improvement could be harvested from changing the thumbnail format alone. The substantially lower file sizes should mean faster thumbnailing and loading due to reduced disk IO and the lower computational complexity of JPG encoding/decoding. We can also investigate further, I tested with JPG because that's what first came to mind.

I should add that, currently, my large thumbnails folder is over 200MB, while I don't recall my Windows thumbnail cache ever exceeding 80MB or so, even with over 1 year of use with no manual cleanup. Will be looking up how they do that.

Is this something you guys are in favor of looking into? I may give it a shot myself, during the week.

(Now on a more off-topic note, can someone confirm that the thumbnail folders are themselves not scanned/processed by the thumbnailer when browsed on? i.e. are they blacklisted?)

Jeremy7701 commented 2 years ago

I run (via cron) a daily command: find ~/.cache/thumbnails/ -type f -mtime +31 -exec rm {} \; [ I should probably run this weekly... ]

This deletes any thumbnails more than 12 days old, which keeps my thumbnails down to about 70MB. Any folders that are not frequently looked at by nemo will get their thumbnails re-generated.

Provided that you don't have gigantic folders, it's probably worth doing something along these lines occasionally as the cached thumbnail doesn't get removed if the file gets moved/edited(?)/deleted etc.

The cached thumbnail folders can be browsed in nemo and viewed with an image program, but even if you request thumbnailing, no thumbnails get created - so in that sense they seem to be blacklisted.

mindinsomnia commented 2 years ago

While this may not be the core factor to the issue being discussed here, I wonder how much improvement could be harvested from changing the thumbnail format alone. The substantially lower file sizes should mean faster thumbnailing and loading due to reduced disk IO and the lower computational complexity of JPG encoding/decoding. We can also investigate further, I tested with JPG because that's what first came to mind.

One downside of JPEG is that it doesn't support transparency. And thumbnails can have transparency backgrounds. One solution would be to use WEBP, it has all the benefits you mentioned of JPEG but with the benefit of transparency support.

I recently switched from using PNG to WEBP for thumbnails on one of my web app projects and the file size difference between the two is very significant.

As an example, here is a picture of a cow that is 1722 x 2073 with a transparent background (subject matter of picture is not relevant for this example).

As a PNG this file is 594.2 kB

16-brown-cow-png-image

As a WEBP this imge is only 92.2 kB with virtually no loss of quality.

(I'd upload the image here but github doesn't support that format yet).

MartinX3 commented 2 years ago

Use AVIF. It's the successor of them all.

And the technical successor of webp.

(VP8 [WebP] -> VP9 -> AV1 [AVIF])

iriki commented 2 years ago

I'm on Windows 10 right now, and if I go to C:\ and search '*', results begin to instantly,

That's because Explorer.exe relies on the "Windows Search" service. You can 100% confirm this by disabling the aforementioned service.

ON-TOPIC When I'm dealing with thumbnails, I don't use Nemo anymore, because of this exact issue. I installed and use thunar and it's much better than ditching Linux Mint ;)

HT-7 commented 2 years ago

Looks like Nemo is trying to determine each file's type by reading their header. This deteriorates performance significantly on non-flash media.

Also, an unresponsive MTP device (e.g. smartphone) can cause Nemo to get very slow. When I click on a directory on any device while the MTP device is connected, I see a blank list for half a minute. No high CPU or I/O usage. No indication of what might be wrong. pkilling and restarting Nemo did not fix it. Only disconnecting the MTP device did.

To its credit, Nemo behaves better here than Windows Explorer. Windows Explorer locks up entirely if an MTP device is unresponsive, and since the task bar and desktop are the same process, they become unresponsive as well.

Jeremy7701 commented 2 years ago

Looks like Nemo is trying to determine each file's type by reading their header. This deteriorates performance significantly on non-flash media.

How else would you do this? The Windows method of deciding a file type entirely from an "extension" is very flakey.

petermoo commented 2 years ago

Is the file type in the Linux file cache? Seems like a good place for it after it is determined.

On 19/6/22 20:39, Jeremy7701 wrote:

Looks like Nemo is trying to determine each file's type by reading their header. This
deteriorates performance significantly on non-flash media.

How else would you do this? The Windows method of deciding a file type entirely from an "extension" is very flakey.

— Reply to this email directly, view it on GitHub https://github.com/linuxmint/nemo/issues/1907#issuecomment-1159684627, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUA7SVDWIZ6KJIW7I6XM5DVP32FJANCNFSM4FLJEVAA. You are receiving this because you commented.Message ID: @.***>

HT-7 commented 2 years ago

Looks like Nemo is trying to determine each file's type by reading their header. This deteriorates performance significantly on non-flash media.

How else would you do this? The Windows method of deciding a file type entirely from an "extension" is very flakey.

The file header checking is indeed helpful and indeed more accurate than that of Windows Explorer, but I suggest having an option for temporarily turning it off to improve performance when necessary.

Jeremy7701 commented 2 years ago

If the mimetype of a file is unknown (which is mostly deduced from the filetype), you won't know what program(s) are capable of processing the file. So switching off this function would be pointless.

If you have a desperate need for speed, switching off thumbnailing via the relevant toolbar button would be more effective if you have a very slow PC.

mindinsomnia commented 2 years ago

If the mimetype of a file is unknown (which is mostly deduced from the filetype), you won't know what program(s) are capable of processing the file. So switching off this function would be pointless.

I think what they meant is: A switch to toggle between determining the filetype by file extension or by reading the file header.

Determining filetype by file extension would be accurate for most types of files, most jpegs have a .jpg extension, most text files have a .txt extension, most html files have a .html extension, etc etc.

Or it could be a progressive enhancement strategy. The filetype could be guessed by file extension initially, and a background thread could determine the mimetype of files more accurate by reading their header in a non-blocking background task.

mindinsomnia commented 2 years ago

Is the file type in the Linux file cache? Seems like a good place for it after it is determined.

Excellent point. If the file's contents haven't changed, no need to reopen it and read the header again to determine it's filetype. And I imagine most times a folder is open, most of the files in the folder have likely not changed between previous visits. Could save a lot of file reads to simply cache the data. The mimetype itself is a relatively small amount of data to store per file.

HT-7 commented 2 years ago

If you have a desperate need for speed, switching off thumbnailing via the relevant toolbar button would be more effective if you have a very slow PC.

I have already deactivated thumbnails, and indeed, the performance is better.

So switching off this function would be pointless.

Sometimes, I don't want to open files, but only manage them (rename, move, etc.). If Nemo did not try to determine file types by header, that would improve performance and reduce wear on a hard disk.

Jeremy7701 commented 2 years ago

You are trying to second-guess what portions of a hard disk that a file system / driver / firmware / hardware is going to access. Not to mention actions of various caching mechanisms, such as at the Linux and hardware levels.

Make sure that you are reading all important files when you are taking your regular backups.

HT-7 commented 2 years ago

Make sure that you are reading all important files when you are taking your regular backups.

Of course I do. Thank you very much.

I have split it into #3020.