Early abandonment of remote package name completion

[kimikage]

As this is related to the tremendous efforts for TTFX, the TTFCompletion of remote package names is still stressful in some cases. (Edited)

The main source of latency (with uncompressed registry) is to check all relevant TOML files for compatibility of candidate packages. You can see what is happening while typing a package name with the following monkey patch:

using Pkg
function Pkg.Registry._parsefile(toml_file::AbstractString)
    println(toml_file) # print file path to stdout
    Base.parsed_toml(toml_file, Pkg.Registry.TOML_CACHE, Pkg.Registry.TOML_LOCK)
end

However, in terms of user input completion, you should be able to give up when the final candidate is no longer determinable. Therefore, I believe that the following monkey patch would be useful.

using Pkg
using Pkg: Registry

REPLExt = Base.get_extension(Pkg, :REPLExt)
JULIA_UUID = REPLExt.JULIA_UUID
function REPLExt.complete_remote_package(partial)
    isempty(partial) && return String[]
    cmp = Set{String}()
    nextchar::Union{Nothing, Char} = nothing
    nextcharidx = length(partial) + 1
    for reg in Registry.reachable_registries()
        for (uuid, regpkg) in reg
            name = regpkg.name
            name in cmp && continue
            if startswith(name, partial) # case-sensitive?
                if length(name) >= nextcharidx # check the next char (In terms of consistency, this might be just before `push!`.)
                   if nextchar === nothing
                      nextchar = name[nextcharidx]
                   elseif nextchar != name[nextcharidx]
                      return String[] # give up
                   end
                end
                pkg = Registry.registry_info(regpkg)
                compat_info = Registry.compat_info(pkg)
                # Filter versions
                for (v, uncompressed_compat) in compat_info
                    Registry.isyanked(pkg, v) && continue
                    # TODO: Filter based on offline mode
                    is_julia_compat = nothing
                    for (pkg_uuid, vspec) in uncompressed_compat
                        if pkg_uuid == JULIA_UUID
                            found_julia_compat = true
                            is_julia_compat = VERSION in vspec
                            is_julia_compat && continue
                        end
                    end
                    # Found a compatible version or compat on julia at all => compatible
                    if is_julia_compat === nothing || is_julia_compat
                        push!(cmp, name)
                        break
                    end
                end
            end
        end
    end
    return sort!(collect(cmp))
end

julia> versioninfo()
Julia Version 1.11.0-beta2
Commit edb3c92d6a (2024-05-29 09:37 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, tigerlake)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)

[kimikage]

Of course, the above is just a PoC, and in practice, separate measures should be taken for cases such as when TAB is pressed twice.

I am not familiar with Pkg or REPL internals, so I am hoping someone will address this issue and backport the measure to v1.11. I showed the monkey patch just in case the improvements were not made in time for the v1.11 release.

[KristofferC]

I was suprised why it would have to go through all the TOML files at all but I see now it is so that it can filter out versions that are incompatible with the Julia version. Maybe that is not worth it...

[kimikage]

Ah, I have been using the uncompressed registry for package development compatible with the older julia (LTS v1.6 or even older). This causes major problems if the compressed registry cannot be recognized for some reason (including the simple case where LTS is installed first).

However, since not all users are using compressed registry and there is still enough latency to be noticeable even with compressed registry (e.g., on hitting "S"), I am certain that this is a design issue.

[KristofferC]

The inline tip is only showed when there is a single matching result so we can just propagate the hint kw from complete_line to complete_remote_package and have it abort as soon as it is more than one matching results.

[kimikage]

This is not directly related to this issue, but I suspect that there are more users using uncompressed repositories than core developers think.

I assumed that compressed (or in-memory) registries would always be used in Julia v1.7 and later, and in fact, I used the compressed registry on Linux without being aware of the default behavior. However, that was because I had separate Linux containers.

[KristofferC]

https://github.com/JuliaLang/Pkg.jl/pull/3913

[kimikage]

Despite my rudeness, thank you for your prompt response.

[kimikage]

just a memo for those who came later: https://github.com/JuliaLang/Pkg.jl/pull/3913#issuecomment-2145313089

I checked the last comment around midnight JST. Pkg precompilation failed on my Windows environment due to a problem with loading LLVM (which should be unrelated to this issue). In the morning, I checked the behavior on Linux. No malice intended there. The only thing is that it is just a little more work to check the stdlib than the usual packages.