What to do with abandonned/unmaintained/orphaned projects?

guyzmo commented 8 years ago

Hi,

I'm posting this issue following an IRC discussion about what would be the policy in case a maintainer of a project is not around anymore (in case they left to become a shepherd in the mountains far from all technology… or when they die, because that happens too).

To quote @dstufft on IRC, that's what the current policy is:

We have a fairly adhoc process that skews heavily towards "if we can't get permission from the original maintainer then nothing happens"

Report a project as orphan?

As a developer and direct user of Pypi's warehouse, I'd be happy to be notified when a given published project is orphan (or likely to be orphan). So it'd be great to have the ability to report that a project is likely to be orphan (with justification for the record, like the date of the last official repository's commit, the number of PR/issues that have been opened but never answered…), so that after proper failsafes against abuses, orphan projects can be spotted easily in pip search, on the project's pypi's page and with a warning on pip install.

Transfer maintainership

As a sole maintainer of a few minor projects on pypi, if at some point I disappear from digital life (which I hope will happen in a very very long time ☺) and I'm unable to maintain or transfer ownership of projects I've been working on alone, I don't want to see them becoming orphans, with nobody able to take them over.

I thought maybe a good idea would be to add in the account preferences a person the maintainer trusts blindly, who could act like an "executor of will".

Then if a project is reported as orphan, and someone wants to take it over as a new maintainer to keep it going, administrators of pypa could contact that person to ask for confirmation that the original maintainer is unable to transfer owership himself, and would give permission to transfer ownership on behalf of the original maintainer.

Other processes/ideas?

I just raised the question as I had to deal with a pypi project that's no longer maintained by its author, and get to wonder about how I would want to see my FLOSS projects (which is somehow my "digital legacy") dealt with.

I don't know how similar projects deal with that (e.g. I know that linux distributions like debian has a way to deal with orphan packages, but pypi mostly contains code directly from the author, unlike debian where author ≠ maintainer). But I have no clue how npm, gem etc. deal with that. if they deal with it at all.

di commented 8 years ago

We probably also need a real policy/process to release a package name that is being "squatted", i.e. has been registered, but hasn't made any releases. For example, what we did in #1487 seems to have worked, but isn't really a formalized process.

But I have no clue how npm, gem etc. deal with that. if they deal with it at all.

Here's how NPM manages package name disputes: https://docs.npmjs.com/misc/disputes

CPAN has a formal process: http://www.cpan.org/misc/cpan-faq.html#How_adopt_module

RubyGems has an "adoption center" that seems to be defunct: https://groups.google.com/forum/#!msg/rubygems-org/niS5ZO9DNgk/5Fhg9Q3QR7YJ

FRidh commented 7 years ago

Relevant is the draft of PEP 541 https://www.python.org/dev/peps/pep-0541/

guyzmo commented 7 years ago

BTW, as a related to this topic, I had a simple idea: it's a common usage pattern that a developer looks up the directory to find a library for his project, and finds several that might match the criteria. Maybe could warehouse provide a "score" that would help decision in terms of activity, responsiveness of the project.

When I have to decide, what I currently do is to lookup all projects, then click on the github link, check for the last commit, if there are issues/PRs when were they last attended by the developer(s), and finally choose the one that looks most active. Maybe I could share that feedback within warehouse using a 👍/👎 scheme.

pradyunsg commented 7 years ago

@guyzmo s/wharehouse/warehouse/ :)

Maybe I could share that feedback within wharehouse using a :+1:/:-1: scheme.

I believe this has been discussed previously and rejected. IIRC, there won't have voting on Warehouse since it is difficult to moderate the same and at the same time it would disadvantage new packages intended to compete with existing established ones.

could wharehouse provide a "score" that would help decision in terms of activity, responsiveness of the project.

:+1: This is worth discussing imo.

Could you check there's is not an issue for something similar and then open a new issue for discussing this idea?

guyzmo commented 7 years ago

Could you check there's is not an issue for something similar and then open a new issue for discussing this idea?

yup there is: cf #991

guyzmo commented 7 years ago

though it's more about a rating, so basically 👍/👎 than really a score that shows how active a project is.

brainwane commented 6 years ago

I suggest we keep this issue to the issue of abandoned, unmaintained, and orphaned projects, and then cover more general and nuanced project activity statistics in #991.

It looks like PEP 541 is on its way to acceptance. I think we can resolve this issue in the following steps:

[x] accept the PEP (this is on the shoulders of the Packaging Working Group)
[ ] Warehouse developers & Packaging WG decide on logistical stuff (what support queue we point users to for PEP 541 requests, #3231/#1919 dependencies)
[ ] Warehouse developers coordinate with the Packaging WG to create the extension to the PyPI Terms of Use and link to it in FAQ and PyPI footer, per PEP requirement
[ ] announce on distutils-sig

brainwane commented 6 years ago

https://mail.python.org/pipermail/distutils-sig/2018-March/032089.html

PEP 541 accepted! @di shall we move forward on the TODOs above? (Am on mobile)

di commented 6 years ago

Yep! Let's figure out our process.

I propose we either figure out https://github.com/pypa/warehouse/issues/3231 or setup pypa/support-requests

Here's the step-by-step process for abandoned projects that I think we should add to our help/policy page somewhere:

1) Identify the project which is invalid/abandoned 2) Open a support request at $LOCATION 3) In the request, demonstrate the criteria for invalid/abandoned projects 4) In the case of an abandoned project, PyPI admins will start the reachability process (6 weeks of attempted contact) 5) PyPI admins resolve the request

dstufft commented 6 years ago

One question we'll need to answer is if these should be public or private requests.

guyzmo commented 6 years ago

about that, I guess it can be both :

use the user's account contact mail first,
then submit an issue to the upstream repository (if it has ticket support, and that task can be automated)

ghost commented 6 years ago

What about deceased maintainers? Is it possible to transfer the maintainership in these cases if a death notice is served?

cc @aaronkaplan

di commented 6 years ago

@wagner-certat Yes, I think that would fall under the category of an "Abandoned Project".

aaronkaplan commented 6 years ago

On 23 Apr 2018, at 16:59, Dustin Ingram notifications@github.com wrote:

@wagner-certat Yes, I think that would fall under the category of an "Abandoned Project".

Thanks Dustin for the answer. What requirements would you have (should we send a certificate of death or so)? To whom?

Best, a.

di commented 6 years ago

@wagner-certat We haven't determined the means by which PEP 541 requests will be created yet, see https://github.com/pypa/warehouse/issues/3231.

Is this a hypothetical issue, or are you attempting to acquire an actual project for which the owner is deceased?

aaronkaplan commented 6 years ago

On 23 Apr 2018, at 17:13, Dustin Ingram notifications@github.com wrote:

@wagner-certat We haven't determined the means by which PEP 541 requests will be created yet, see #3231.

Is this a hypothetical issue, or are you attempting to acquire an actual project for which the owner is deceased?

The latter. It's real unfortunately. I am one of the co-authors of mihi's https://github.com/FFM/pycryptopan project.

I have full write access to the github repo. However, mihi died some years ago and he was the single person responsible for uploading it to pypi (https://pypi.org/project/pycryptopan/).

As a matter of virtual courtesy, I'd like to make sure that the code gets improved and actually made available for the world - on pypi.org - after all it's a pretty useful small code project which can be used to anonymise a lot in times of GDPR.

So, yes, I would like to be able to upload new code to https://pypi.org/project/pycryptopan/ But for that account, only Mihi has access.

I am also in contact with his left behind relatives who could help me with formalities in case you require that.

Best, Aaron Kaplan.

di commented 6 years ago

@aaronkaplan Thanks for the clarification. I'd suggest that you make a separate issue to capture this request. We're currently still defining the process for PEP 541, and once we do we have a large backlog to work through as well, so please have patience.

aaronkaplan commented 6 years ago

On 23 Apr 2018, at 17:36, Dustin Ingram notifications@github.com wrote:

@aaronkaplan Thanks for the clarification. I'd suggest that you make a separate issue to capture this request. We're currently still defining the process for PEP 541, and once we do we have a large backlog to work through as well, so please have patience.

ok. can of course make a separate issue. What time frame are we looking at? Months? Weeks? Days?

Best, a.

aaronkaplan commented 6 years ago

okay, separate ticket #3792 created. Please advise on further steps.

brainwane commented 6 years ago

(Along the way, someone ought to update the draft "Procedure for Transferring PyPI Name Ownership" Google doc because evidently some people arrive at that still thinking it's the most current process documentation.)

vidstige commented 6 years ago

@di @brainwane do we really need the Warehouse support tool (#3231) in order to start processing the backlog of abandoned package claims? Now with PEP-541 finally accepted, the paperwork is in place to process the backlog.

I feel this is currently more of process problem rather than a technical problem: Who will process the support requests? The answer to this question will be no less clear with #3231.

Processing this type of support tickets should take fairly small amount of time per ticket, but will take a lot of calendar time (waiting for eventual answers from current owners). This, in conjunction with the already dire backlog, lends me to believe the community would benefit more from action rather than waiting for more Warehouse features.

nsheff commented 6 years ago

Just wanted to add a thought here... have you thought at all about introducing a type of namespace for pypi packages?

With general increase in programming literacy and package development and python's continued popularity growth, package name conflicts are likely to only increase, probably dramatically, over the next few years. Maybe it's worth thinking about employing the method used by github, dockerhub, and others to make the package index use namespaces (so it would be pip install username/packagename). Maybe packages that reach a certain utility (measured by downloads or something) could be granted a spot in the top-level namespace. it seems there are a lot of old, empty, abandoned, or community useless packages consuming high-value top-level names at the moment, and this will get worse.

anarcat commented 6 years ago

@nsheff that's a good idea, but in the end that only moves the goalposts around. e.g. on GitHub there are "organizations" that have similar problems than any other toplevel nomenclature. i had to deal with this for the Linkchecker project and the only reason it was easier than on PyPI is that the maintainer on GitHub was different than PyPI: they were responsive and collaborated with the transition.

In other words, while it might, at first look, seem like a nice solution, it won't actually fix the problem for shared namespaces.

Really, at this point I think we just need to figure out who takes care of the support requests. The PEP 541 label in pypi-legacy is probably fine to start with: there are less than 20 projects to deal with right now, and it will probably take longer to discuss a ticketing system and implement it than to just work through those tickets...

For the record, PEP541 says that the "Package Index maintainers" are basically responsible for dealing with such issues, but can escalate to the PSF board which makes recommendations to the Packaging WG who has the final say. But hopefully, most pending issues can just be dispatched easily by the maintainers at this stage. I know that in my case, it's a fairly clear-cut scenario that shouldn't need much more discussion, and I suspect it's probably most of the cases.

I would also suggest adding issue templates to make sure people have gone through all the steps in the process before filing issues, before wasting the maintainers time. ;)

nsheff commented 6 years ago

@anarcat you're right that it only moves the goalposts around... but it does move them a lot. if the current, one-namespace method gives you n possible top-level package names, then the github two-level method gives you n^n, which, since n is quite large, is a big difference in space for "good" names... even though it won't solve name conflicts completely, it would dramatically reduce them.

anyway, you're probably right that it's not really necessary at this point... and the other changes will surely improve things, as you say. So, maybe pypi isn't big enough yet for this to make sense. BUT, I'm just wondering if this will become necessary in the future. you can probably find some repository names with dozens or hundreds of duplicates on github (in different user spaces) -- one way to think of it is that github outgrew the 'one-layer' strategy many years ago. pypi packages will never reach the number of github repositories, but two layers is a viable strategy for dealing with name duplication at the scale of github, so that's quite a big goalpost change. maybe that's something to think about as pypi gets more saturated. maybe not this year, but what about next year, or in 5 years? Might be worth thinking about future-proofing things.

mortenlj commented 6 years ago

@nsheff : While I'll agree that your idea has merit, I don't think it is a requirement before deciding on a process and handling the backlog of PEP541 transfers?

I don't see a conflict between deciding on a process and taking care of the PEP541 backlog while discussing/designing a namespace solution for future improvement. I think namespaces for packages could/should be discussed in a separate issue, where it can be the sole focus of the discussion, instead of piggybacking on an almost solved, slightly related issue.

Disclaimer: I'm not involved in pypi development, I'm just waiting for a PEP541 transfer...

pradyunsg commented 6 years ago

Hey @mortenlj @nsheff @anarcat!

There's #2589 for introducing namespaces for packages. I'd appreciate if you could move the discussion about the it there; since this issue is distinct from that one and likely the best place for PEP 541 updates.

mfrasca commented 6 years ago

The PEP 541 label in pypi-legacy is probably fine to start with: there are less than 20 projects to deal with right now, and it will probably take longer to discuss a ticketing system […]

could it be, that the list is short, because not many people were aware of this possibility?

nsheff commented 6 years ago

There's #2589 for introducing namespaces for packages. I'd appreciate if you could move the discussion about the it there; since this issue is distinct from that one and likely the best place for PEP 541 updates.

Ah, perfect! sorry to hijack the thread.

dialex commented 5 years ago

FYI this might help: https://github.com/rejuvenate/rejuvenate

LalamN commented 5 years ago

In a worst case,if the github platform is abandoned and shutdown permanately without selling.what happens to our projects and repositories does they have any backups, if not how they can be retrieved?

di commented 4 years ago

Calling this closed. The answer is defined in PEP 541.

brainwane commented 4 years ago

It looks like PEP 541 is on its way to acceptance. I think we can resolve this issue in the following steps:

* [x]  accept the PEP (this is on the shoulders of the [Packaging Working Group](https://wiki.python.org/psf/PackagingWG))

* [ ]  Warehouse developers & Packaging WG decide on logistical stuff (what support queue we point users to for PEP 541 requests, #3231/#1919 dependencies)

* [ ]  Warehouse developers coordinate with the Packaging WG to create the extension to the PyPI Terms of Use and link to it in FAQ and PyPI footer, per PEP [requirement](https://github.com/python/peps/blame/master/pep-0541.txt#L84)

* [ ]  announce on distutils-sig

@di what about these remaining TODOs, though?

di commented 4 years ago

Sorry, missed those.

DriesSchaumont commented 3 years ago

Hi! Is #3231 blocking this issue? If so, could I propose that a temporary process for PEP541 is formalized, while #3231 is being resolved? This way PEP541 requests can still move forward. I noticed that the current best-practice is to open an issue at https://github.com/pypa/pypi-support, but waiting for #3231 might cause larger backlogs than desirable.

Thanks in advance! Dries

toonarmycaptain commented 3 years ago

May I suggest some sort of addition to this process, where the names of python2 only projects be made available to new projects on PyPI? I'm looking at a package whose last commits were in 2013, the package never supported Python3 (import yields a relative import error in Python3).

Obviously these names might be offered to the original maintainers with the caveat that the projects get updated, but, failing this, they should be made available?

pypi / warehouse