Pre-compute / Cache Recursively & Eliminate Repetitive Computation

emrakyz commented 1 month ago

`yazi --debug` output

Yazi
    Version: 0.3.3 (VERGEN_IDEMPOTENT_OUTPUT 2024-09-10)
    Debug  : false
    OS     : linux-x86_64 (unix)

Ya
    Version: 0.3.3 (VERGEN_IDEMPOTENT_OUTPUT 2024-09-10)

Emulator
    Emulator.via_env: ("foot", "")
    Emulator.via_csi: Ok(Foot)
    Emulator.detect : Foot

Adapter
    Adapter.matches: Sixel

Desktop
    XDG_SESSION_TYPE           : Some("wayland")
    WAYLAND_DISPLAY            : Some("wayland-1")
    DISPLAY                    : Some(":0")
    SWAYSOCK                   : None
    HYPRLAND_INSTANCE_SIGNATURE: Some("7c4c402bd7f24241c096d809cc80408a469f15cb_1725955248_86907018")
    WAYFIRE_SOCKET             : None

SSH
    shared.in_ssh_connection: false

WSL
    WSL: false

Variables
    SHELL              : Some("/bin/zsh")
    EDITOR             : Some("nvim")
    VISUAL             : None
    YAZI_FILE_ONE      : None
    YAZI_CONFIG_HOME   : None

Text Opener
    default: Some(Opener { run: "nvim \"$@\"", block: true, orphan: false, desc: "$EDITOR", for_: None, spread: true })
    block  : Some(Opener { run: "nvim \"$@\"", block: true, orphan: false, desc: "$EDITOR", for_: None, spread: true })

Multiplexers
    TMUX               : false
    tmux version       : No such file or directory (os error 2)
    ZELLIJ_SESSION_NAME: None
    Zellij version     : No such file or directory (os error 2)

Dependencies
    file             : 5.45
    ueberzugpp       : No such file or directory (os error 2)
    ffmpegthumbnailer: 2.2.2
    magick           : 7.1.1-38
    fzf              : 0.55.0
    fd               : 10.2.0
    rg               : 14.1.0
    chafa            : No such file or directory (os error 2)
    zoxide           : 0.9.4
    7z               : No such file or directory (os error 2)
    7zz              : 24.08
    jq               : 1.7.1

Please describe the problem you're trying to solve

I am on Gentoo Linux and I have a high-end PC (32 threads). I can't browse my external HDD properly using Yazi. Any other file manager is lightning fast.

My HDD is an 8TB, external, slow, USB 3.0 external disk mounted on "/mnt/dir". The directory has more than 500.000 files in a nested structure.

This directory is owned by the one and only current user and is mounted at boot.

The problem is that Yazi can't use the cache properly and there is no way I could find to pre-compute each of the 500.000 files in my HDD without manually traversing and save the data in a cache before opening the file manager. For example, on LF, this HDD is completely cached and I never ever need to do a computation (unless I add a new file). Even then, LF computes things on the fly (when you are on an entry), and it doesn't bother the device unnecessarily (even though this is technically worse).

Yazi tries to compute to find mime-types and create previews when I try to browse; every time from scratch. Since the drive is slow, I at least need to wait for a minute in order to open a video for example. And even if I can open a video; I can't play it properly because of stutters (as there would be unfinished background tasks going on continuously).

At the same time, I have lots of directories and files in that disk and as they are dynamically showing up one by one, I can't see most of my directories and/or files. Therefore, it confuses me. I need to be able to see all of them immediately and I need to be able to read/write without waiting for anything or without spending computational resources unnecessarily.

I have the below setting in "${XDG_CONFIG_HOME}/yazi/yazi.toml":

[preview]
cache_dir = "${XDG_CACHE_HOME}/yazi-1000"

I confirm that this cache directory and its content are created and preserved across reboots.

I guess the cache functionality works for my home directory but not external devices on /mnt. Everytime, I see things like that (see the below image).

To sum up, I have two problems that make me use an alternative file manager whenever I want to access my external device:

There is no way to pre-compute for all files in a device.
Even the already computed part is re-computed again and again; despite having the cache directory inside "${XDG_CACHE_HOME}" instead of /tmp.

Would you be willing to contribute this feature?

[ ] Yes, I'll give it a shot

Describe the solution you'd like

Provide an optional CLI flag in order to recursively traverse all directories and pre-compute & cache: Mime types, directory structure, metadata, image and other previews, so the user won't have to do the computation again and again; especially for bigger and slower disks.

Such as: $ yazi --cache "/mnt/dir" Waits for ~10 minutes... Done

Even without this flag; the cache should always be prioritized. Right now, the modified cache_dir works with "${HOME}". I can open Yazi on my home directory, and there is no computation in the background since I already have the cache. But on my external HDD, even the directories I manually traversed are not saved across multiple instances.

Additional context

No response

Validations

[X] I have searched the existing issues
[X] The latest nightly build of Yazi doesn't already have this feature

sxyazi commented 1 month ago

From the screenshot, these are mime tasks - it's a fetcher used to retrieve file metadata, so it's unrelated to caching as they're not creating caches at all. In your case, you need the https://github.com/yazi-rs/plugins/tree/main/mime-ext.yazi plugin.

BTW in case you're having issues with pre-caching as well, Yazi offers the ability to completely disable preloads:

# ~/.config/yazi/yazi.toml
[plugin]
preloaders = []

These instructions are already included in the tips, please take a look at https://yazi-rs.github.io/docs/tips#make-yazi-even-faster

emrakyz commented 1 month ago

@sxyazi

Thanks for the answer.

I had already read the wiki. I simply forgot about that plug-in.

Yes. This makes everything better but my actual request was not that.

I don't want to disable image/video previews. It's the opposite. I need a way to pre-compute everything for the previews. Now, the preview cache works but yet, I don't always browse the same directories. There are tons of them.

Instead I want to be able to pre-compute them because I have 500.000+ files in that hard drive and I don't want to do the computation manually by traversing all directories because Yazi only does the computation for the close proximity which is logical but taxing for these kinds of situations.

A functionality like my below example, would be very cool. Or maybe you can give me an idea in order for me to write a script to traverse directories slowly with Yazi by waiting for all previews to be loaded and cached one by one. Is there a similar way or a possibility?:

$ yazi --cache "/mnt/dir"
Waits for ~10 minutes...
Done

sxyazi commented 1 month ago

I don't want to disable image/video previews

No, preloaders = [] does not disable image or video previews. It only disables preloading, meaning that files will only start being read and processed when you actually move the cursor over them to preview like other file managers do, instead of loading them as soon as you open the directory.

Note that preloaders and previewers are two different concepts, and previews are disabled only if previewers = [] is set:

https://github.com/sxyazi/yazi/blob/61c0db8630bf48ff3fd3919f736704837f50f7f5/yazi-config/preset/yazi.toml#L95

https://github.com/sxyazi/yazi/blob/61c0db8630bf48ff3fd3919f736704837f50f7f5/yazi-config/preset/yazi.toml#L107

emrakyz commented 1 month ago

@sxyazi

Oh, thank you. That makes sense. These are different concepts.

What about pre-caching in batch? Is there a native way? Or should I write a script by spawning terminals with Yazi for each directory in the target location and simulate key-presses with ydotool in order to traverse every part of a directory?

My main aim is to have the cached image/video previews inside the cache_dir so there won't be any waiting for the computation.

sxyazi commented 1 month ago

I don't think bulk caching is the best way to solve this problem, given the size of your files, it could take a long time to generate and occupy a lot of space. Plus, you might need to repeat this process regularly to remove invalid caches (for example, when file contents change or files no longer exist) and create new incremental caches based on the updated file tree.

However, if you really need it, I can consider exposing the preload() methods for the image and video preloaders as a Lua API, so you can use a plugin to call it caching these files.

github-actions[bot] commented 4 weeks ago

I'm going to lock this issue because it has been closed for 30 days. ⏳ This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

sxyazi / yazi