Add ability for packages to be "unlisted" in registry

[oxinabox]

Idea: We should add a field in the registry for to mark packages as "unlisted". This can be applied to abandoned packages, or to packages that gave been deprecated or renamed.

The effect of this woud be to make it so that pkg> add ... REPL does not tab-complete to that name. Possibly some other things might like to use it, e.g. JuliaObserver, or JuliaHub might do something with it.

To go beyond this, perhaps we could have a free text field that the unlisting reason could be given. Which would be displayed when people directly call pkg> add. And that could give a warning message explaing e.g. that it was renamed and that other one should be installed instead, or that it has been discontinued and a good alternative might be... Such a message would not be displayed when it is added indirectly.

Naming things is hard, so perhaps this should be listed = false or visiblity = supressed, or even unlisted-message = "Reason this is unlisted" and if not present then it is listed. or something.

[StefanKarpinski]

I like listed = false but we could also have something like status = "deprecated".

[KristofferC]

Maybe deprecated = true.

[oxinabox]

Another use of this is that registry CI should ignoring unlisted entries for purposes of checking for too similar names.

Maybe even if two packages are registered with the same name but one is unlisted it should not present the menu to chose them

[StefanKarpinski]

I think all of that falls under the concept of "deprecated". Although at that point, it feels like we may want to just split deprecated packages into a Deprecated registry. I believe that any client using the pkg protocol will install all the registries that the pkg server advertises, so we could introduce more registries pretty smoothly.

[oxinabox]

Is it ok that packages in a Deprecated registry would depend on packages in the General registry and packages in the General registry would depend on packages in the Deprecated registry? I think that works fine?

---

One problem i have with the word deprecated is to me it implies there is a replacement. "X is deprecated in favour of Y". But for some things they might be unlisted without replacement. Like if everyone decided it was a terrible idea. Or if something is super-buggy and the author admits that, but has no time or interest in fixing it.

[KristofferC]

I don't think a deprecation necessarily implies an existing replacement (although that is common).

Although at that point, it feels like we may want to just split deprecated packages into a Deprecated registry.

Then it would have to be installed by default? Or otherwise you can't instantiate a manfiest with a deprecated package.

[Nosferican]

PyPI uses:

• Inactive

CRAN uses:

• archived
• orphan (unmaintained)

[MasonProtter]

I think this would be really good to have. We can allow people to install these packages, but print a big warning whenever they do, and repeat the warning every time they attempt to update their packages. Something like e.g.

┌ Warning: DataFitting.jl is a deprecated package, it is unmaintained 
| and installing it will cause version conflicts which prevent you from
| installing modern versions of other packages.
└ @ Main REPL[7]:1

[MasonProtter]

An idea raised by @giordano in https://github.com/JuliaRegistries/General/pull/38626#issuecomment-864627898 was to just cap the julia compat version for packages like this so that the offending package becomes invisible on newer versions of julia.

That seems like a pretty elegant solution that also doesn't need new functionality to be added.

[StefanKarpinski]

That's not great though because you might be trying to run a manifest with a new Julia version and it would work but doesn't only because of fake compat caps. More abstractly, introducing misinformation into the registry doesn't seem ideal: deprecated/unlisted packages might still work, so claiming they don't is just a lie.

[KeithWM]

For me this is a becoming an ever more urgent requirement to keep the Julia language spreading. As time passes, the number of abandonned packages is growing. It is a real pain for newcomers to have to figure out which of the n packages that claim to do x is the relevant one.

But with whom does the decision lie to deprecate an entire package? I think leaving it up to the package owners is a poor choice, as they have often abandonned the project.

For me, a warning would suffice, preferrably pointing to the new package, of course. I think we get into dangerous territory if we start allowing name clashes.

[oxinabox]

Process is not needed to be decided here. This is the Pkg issue for the feature But implictly because the information is stored in the registry it is a registry maintainer. Who would likely merge such a PR if it is from the packages maintainer, or if it is well justified from anyone in the community. (after all we can always undo that change)

[ronisbr]

One thing that might help here (at least for newcomers) is to improve the experience when searching for packages. I am not sure how hard it will be to populate the data, but there should be a way to search for packages within REPL. AFAIK, we can only perform this kind of searching online. In REPL, we only have the autocompletion which uses only the package name.

Hence, when the user searches for something, they can receive a nice table with the package name, package version, description, and state. The latter would be used for the maintainer to mark the package "abandoned" or something else. Something like this:

(@v1.10) pkg> search SatelliteToolboxT*
 Package                             Version              Description                                             Status
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 SatelliteToolboxTle.jl              v1.0.6 (2024-05-22)  Creating, fetching, and parsing TLEs.                   Active
 SatelliteToolboxTransformations.jl  v0.1.8 (2024-05-24)  Transformations for the SatelliteToolbox.jl ecosystem.  Active

Of course, the search should also work on the description field, leading to a better discoverability of the packages.

EDIT: I thought the desc field on Project.toml is copied to the general registry, but it is not. Hence, this feature would require some modifications.

[KeithWM]

I've never used Pkg's search functionality, nor have I heard other mention it, so I'm not sure how relevant it is.

I often find myself pointed to packages via Google or to solutions on Discourse that later appear to have been superceded by a newer package. And it's not like these are really old posts either.

@oxinabox Indeed the process doesn't have to be defined here, but the solution should be something that also works if the package maintainers are no longer available, willing or able to cooperate.

[ronisbr]

I've never used Pkg's search functionality, nor have I heard other mention it, so I'm not sure how relevant it is.

There isn't :) My proposal is to implement one so that we can see the current state of the package, including the last version release date and the state.

[mitiemann]

Would it be possible to get and display update and download statistics for interested users? I would assume that this would often give a good enough indication whether a package is maintained or not. It would also solve the problem of finding the de facto standard package in certain domains. E.g., which AD package is the most common.