Inconsistent result order

[lervag]

Description

When I use find_files to list a directory with many files, the order is inconsistent between each consecutive use. The same if I apply a filter; results are still inconsistent between different runs with the same filter.

Neovim version

NVIM v0.8.0-dev+226-g07ade91f21
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3

Operating system and version

Arch Linux

checkhealth telescope

telescope: require("telescope.health").check()
========================================================================
## Checking for required plugins
  - OK: plenary installed.
  - OK: nvim-treesitter installed.

## Checking external dependencies
  - OK: rg: found ripgrep 13.0.0
  - OK: fd: found fd 8.3.2

Steps to reproduce

Given the below test.vim: Run two terminals, in both, go to directory with a lot of files and do: nvim --clean -u test.vim. Compare results.

Expected behavior

Consistent set of results.

Actual behavior

Inconsistent set of results.

Minimal config

" test.vim
set runtimepath^=~/.local/plugged/plenary.nvim
set runtimepath^=~/.local/plugged/telescope.nvim

lua <<EOF
require 'telescope'.setup {
  defaults = {
    tiebreak = function(_current_entry, _existing_entry, prompt)
      return false
    end,
  },
}
EOF

inoremap q <cmd>qall!<cr>

autocmd VimEnter * Telescope find_files

[fdschmidt93]

To clarify, this affects initial sorting of files?

I'm pretty sure that this depends on your provider of find_files, i.e. fd, rg, etc. as opposed to telescope.nvim. Most likely, they will run multi-threaded (default) and order therefore is not guaranteed for the sake of being fast.

Quick glance over fd issues, not even a single thread doesn't guarantee ordering but requires --list-details (as per man fd) instead. Single-threaded IIRC makes order deterministic for rg. If you'd need a deterministic order, I suppose ideally you'd have an option from your provider (tiebreaker will be too slow I guess at least for initial sorting, ie breaking all the ties to eg sort alphabetically).

[lervag]

Thanks for the quick reply!

To clarify, this affects initial sorting of files?

Yes, probably. I don't understand the mechanics of Telescope well enough to be sure yet.

I'm pretty sure that this depends on your provider of find_files, i.e. fd, rg, etc. as opposed to telescope.nvim. Most likely, they will run multi-threaded (default) and order therefore is not guaranteed for the sake of being fast.

I believe fd is the default if it is available. And yes, you are right that the result order of fd itself is inconsistent. So this may be the reason for my issue.

Quick glance over fd issues, not even a single thread doesn't guarantee ordering but requires --list-details (as per man fd) instead. Single-threaded IIRC makes order deterministic for rg. If you'd need a deterministic order, I suppose ideally you'd have an option from your provider (tiebreaker will be too slow I guess at least for initial sorting, ie breaking all the ties to eg sort alphabetically).

Ok; so, if I understand correctly, you are saying that I may get what I want by changing the find_files provides to a tool that can give deterministic order, and that rg is such a tool if I specify to run single-threaded.

I tried this, and I find that e.g.

find_command = { "rg", "--files", "--sort", "path" },

works as desired. I.e., the results are now in deterministic order.

I guess this mostly leaves my issue resolved! Thanks!

Just out of curiosity: Do I understand correctly that the mechanism called "sorter" in Telescope does not necessarily sort the results? E.g., fzf-native only applies filtering, not sorting. Or, that is only party true: fzf-native sorts the results according to scores based on the query. Candidates that does not matched are filtered out, and the other results are sorted to score order. Please correct me if I am wrong. If not: Would it be interesting or useful to add something like a "pre sorter" or "pre filter"? Or does it not make sense, as that is something that should be left to the candidate source?

[fdschmidt93]

Apologies for practically replying twice to your issue (here and Matrix/Gitter); for some reason I only got the notification from Element 11 hours later, overlooked the time stamp of your question there, and then confused proceeded to reply.

Do I understand correctly that the mechanism called "sorter" in Telescope does not necessarily sort the results?

telescope.nvim in conjunction fzf-native returns the original results in order as they arrive if there is no prompt and then bypasses tiebreak, since no prompt always returns a score of 1 (which I believe is the upper threshold and worst match, with 0 being lower bound and best match) and the tiebreak being only triggered if score is smaller than 1 and scores of two entries are equal. In order words, there is no signal as to how to sort them (well, you could make the argument tiebreak is a signal, but breaks ties from actual sorting). Default tiebreak prefers shorter strings (I think like fzf). In other words, entries are only sorted once the user puts in a query (i.e. prompt).

If not: Would it be interesting or useful to add something like a "pre sorter" or "pre filter"? Or does it not make sense, as that is something that should be left to the candidate source?

Something like this already sort of exists; try LSP Document Symbols and hit <C-l> - completion should pop up allowing you to iterate (ie pre-filter) through varying types of symbols of your buffer. While I suppose users could lever this already, this is quite a stoic framework for now. We have plans to eventually generalize the concept of "filtering" into being able to chain an arbitrary number of (possibly user tailored) sorters.

Hope that answers your questions!

[lervag]

Thank you for the very clarifying and good response!

I'm loving Telescope so far, by the way. Great work - thanks to you and everyone else who've made this! <3