Performance Issues with Large Number of Images

[felipesanmartin]

Description:

I am experiencing significant performance issues with the cosmic-files application when handling directories containing a large number of images. Specifically, I have observed the following problems:

Slow Runtime and High Memory Consumption:

When processing directories containing approximately 30,000 images, the application exhibits slow runtime and high memory usage. With directories containing around 100,000 images, the application fails to function properly and does not complete the operation. Possible Cause:

It appears that cosmic-files attempts to load information for all subdirectories and generate thumbnails for all images simultaneously. This approach may be leading to excessive memory usage and performance degradation.

Steps to Reproduce:

Place approximately 30,000 images into a single directory. Attempt to process this directory using cosmic-files. Observe the slow runtime and high memory consumption. (For more severe issues) Place around 100,000 images into a directory and attempt to process it. Expected Behavior:

The application should efficiently handle directories with a large number of images without significant performance degradation or excessive memory usage. Ideally, it should process images in a more scalable manner, potentially by using lazy loading or batch processing.

Actual Behavior:

The application becomes very slow and consumes a high amount of memory when processing large directories. When dealing with extremely large directories (e.g., 100,000 images), the application fails to operate as expected.

Proposed Solution:

Consider implementing optimizations such as:

Lazy Loading: Only load and process image data or thumbnails when they are actually needed. Optional Thumbnail Display: Provide an option to disable thumbnail generation or display in list views, which can help reduce memory usage and processing time for large directories.

Additional Information:

Version: epoch-1.0.0-alpha.1 Environment: Pop!_OS 24.04

Hardware Specifications: CPU: Intel Core i7-11700F RAM: 32 GB Disk: 512 GB SSD Graphics Card: NVIDIA RTX 3060

Please let me know if further information is needed or if there are any workarounds available.

Thank you for your attention to this issue.

[leviport]

I think this might be a duplicate of https://github.com/pop-os/cosmic-files/issues/268? Unless it's specifically worse with image files.