Feature request: support the compressed (non-extracted) version of the Pkg server registry

[CameronBieganek]

julia> all_pkg_names = AutoMerge.get_all_non_jll_package_names("/Users/cameron/.julia/registries")
ERROR: "/Users/cameron/.julia/registries/Registry.toml": No such file
Stacktrace:
 [1] error(::String, ::String)
   @ Base ./error.jl:42
 [2] readstring
   @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/TOML/src/TOML.jl:21 [inlined]
 [3] parsefile(f::String)
   @ TOML /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/TOML/src/TOML.jl:43
 [4] get_all_non_jll_package_names(registry_dir::String)
   @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/Nj1Mv/src/AutoMerge/util.jl:285
 [5] top-level scope
   @ REPL[7]:1

Here are the contents of my ~/.julia/registries directory:

total 4.2M
-rw-r--r-- 1 cameron staff 4.2M Feb 27 20:51 General.tar.gz
-rw-r--r-- 1 cameron staff  129 Feb 27 20:51 General.toml

[DilumAluthge]

There are three possible formats that the registry can take. Unfortunately, RegistryCI currently only supports two of them.

Git or Pkg server	Tarball has been extracted	Supported by RegistryCI	JULIA_PKG_SERVER	JULIA_PKG_UNPACK_REGISTRY
Git		Yes	ENV["JULIA_PKG_SERVER"] = ""	delete!(ENV, "JULIA_PKG_UNPACK_REGISTRY")
Pkg server	Yes	Yes	delete!(ENV, "JULIA_PKG_SERVER")	ENV["JULIA_PKG_UNPACK_REGISTRY"] = "true"
Pkg server	No	No	delete!(ENV, "JULIA_PKG_SERVER")	delete!(ENV, "JULIA_PKG_UNPACK_REGISTRY")

To switch your registry format:

1. import Pkg
2. Pkg.Registry.rm("General")
3. Run the approprate commands to set/unset the environment variables shown in the table above.
4. Pkg.Registry.add("General")

For example, to switch to the Pkg server extracted registry:

import Pkg

Pkg.Registry.rm("General")

delete!(ENV, "JULIA_PKG_SERVER")
ENV["JULIA_PKG_UNPACK_REGISTRY"] = "true"

Pkg.Registry.add("General")

[DilumAluthge]

@CameronBieganek Can you confirm that switching to one of the two supported registry formats fixes the problem for you? If so, then we can change this issue to be a feature request for RegistryCI to support the non-extracted Pkg server registry. But let's first make sure you can get things working.

[CameronBieganek]

Are there any performance advantages to sticking with the unextracted tarball? If so, I'd rather stick with that. Although I suppose I could probably just delete the registry and reinstall it if I want to go back to an unextracted tarball. This isn't an important issue for me---I just wanted to check out the string distance for a package name.

[DilumAluthge]

On most systems, it is more performant to use the unextracted tarball.

You could just clone the General registry to a temp directory, and pass that directory as the first positional argument to get_all_non_jll_package_names.

[CameronBieganek]

Cool, thanks! I'll try that when I get a chance.

[ericphanson]

I think the way to do this without needing an unpacked registry would be for someone to help setup https://github.com/GunnarFarneback/RegistryInstances.jl as a stable face in front of Pkg's registry-handling internals on Julia 1.7+, and then use it here.

[GunnarFarneback]

That package is stuck because I haven't needed it badly enough myself yet. The major missing part is to extract the relevant tests from Pkg and if necessary add some new ones.

[cgarling]

Just adding that I hit this today and would be interested in this feature. Thanks for your work on the CI infrastructure!