search

haskellfoundation / stability

Issues and proposals related to the HF Stability Working Group
25 stars 2 forks source link

Introduce Haskell Party proposal #12

Open bergmark opened 2 years ago

bergmark commented 2 years ago

I have been thinking about two related things when it comes to package maintenance, how can we:

The Haskell Foundation Stability Working is discussing these topics and I have written a proposal that I think will be easy to implement and not conflict with any of our ongoing work.

Any feedback is appreciated. Do you think this would be useful? Would you like to help out? Do you have any Haskell projects that would fit in here?

Feel free to post your comments here or on Discourse.

hasufell commented 2 years ago

I think you've identified some pain points of the community, but after reading through the proposal, it is not entirely clear to me what joining it means for me as a package author.

E.g. the update process through hackage trustees has a very narrow pre-defined scope: only patches that make the project buildable with newer GHC versions are allowed. Feature patches or even general bugfixes are not.

What is the scope of changes that may go through unapproved by the package author through the Haskell Party maintenance system? I'd imagine authors being worried that design decisions are being upheld, even in their absence. The hackage trustee process kind of ensures that this doesn't happen, unless a proper package takeover happened (in which case trustees will try to repeatedly contact the author beforehand).

However, I can see the benefit of a centralized community place where authors or end-users of libraries ask for help wrt maintenance issues (instead of 100+ individual bugtrackers).

My opinion on how to fix these maintenance issues is this:

  1. Drive forward https://github.com/haskellfoundation/stability/issues/7 (the working group said they don't have bandwidth for this, which left me rather disappointed, since I think it was the first proposal that's immediately actionable and may have very high impact): this will encourage maintainers to look for backup maintainers simply to be featured as a "well-maintained" project.
  2. Implement namespaced packages in hackage https://github.com/haskell/hackage-server/issues/924: this will make forking packages on hackage a bliss, so you can switch to a fork with very low overhead. Hackage could additionally allow to show all forks of a package, so you can get an overview of your options.
  3. Have a central place, where users and authors can discuss maintenance issues, advertise calls for help, organize GHC mass updates (these currently happen implicitly in stackage issue tracker and on head.hackage). I'm not sure if Haskell Party is exactly this. It may also be possible that this is implemented directly in hackage as a meta-issue tracker, but it may be overkill
    • Additionally, we could make head.hackage be more ergonomic both for end-users and contributors. I think it's currently only really used by GHC devs.
silky commented 2 years ago

I haven't seen it mentioned, but this reminds me of this organisation - https://github.com/haskell-pkg-janitors - which I guess had a similar aim; but ended up being quite inactive.

chshersh commented 2 years ago

This proposal looks the closest to how I see the solution to delayed updates of packages in the Haskell ecosystem. And I think the proposal can be simplified (in my vision) to the following:

That's it. There're some details that need to be specified (e.g. activity expectations from maintainers, how to maintain the list, etc.). But in my vision, the main reason behind the inability to update packages quickly is that all packages are managed by volunteers in their free time.

They don't have to upgrade their packages quickly and nothing can force them to do this.

But if they add other people to their projects and grant them merging and Hackage uploading rights, package upgrades can happen without the need to wait for the maintainer. A dozen of volunteers can migrate the entire ecosystem to a new GHC within a couple of weeks.

I don't think a separate organization for this is required. When the maintainer is MIA, someone from the stability group can fork their repo (to their personal GitHub account or even better to the haskell org) and update maintainership information.

bergmark commented 2 years ago

Thank you for the feedback @hasufell

after reading through the proposal, it is not entirely clear to me what joining it means for me as a package author.

I agree that this is hard to discern. I will try to restructure this to make it clearer.

What is the scope of changes that may go through unapproved by the package author through the Haskell Party maintenance system?

As it's written (or at least, as I was thinking when I wrote it...), we'd try to find maintainers that can help out and it would be up to the author to decide what they would like help with, although we can't enforce this technically. I imagine some maintainers saying "I only want help with patches to support new GHCs and dependencies" and some saying "Do whatever you want". It's up to the author how much control they want. Perhaps we can create a form for this.

I'd imagine authors being worried that design decisions are being upheld, even in their absence. The hackage trustee process kind of ensures that this doesn't happen

I think the key here is this is all opt-in, and it will be based on trust. If this sounds scary, perhaps we can clarify the process. Or the author can wait a bit and see how the project progresses, perhaps seeing that this is not a problem for the packages we host would help?

Drive forward Maintainership standards #7 (the working group said they don't have bandwidth for this, which left me rather disappointed, since I think it was the first proposal that's immediately actionable and may have very high impact): this will encourage maintainers to look for backup maintainers simply to be featured as a "well-maintained" project.

I think we set this aside for a reasons:

Implement namespaced packages in hackage Namespacing packages haskell/hackage-server#924: this will make forking packages on hackage a bliss, so you can switch to a fork with very low overhead. Hackage could additionally allow to show all forks of a package, so you can get an overview of your options.

I think this would be very nice, and it is necessary at times. But ideally there would be no fork, just one well-maintained library, right? This is what this proposal is trying to help with.

Additionally, we could make head.hackage be more ergonomic both for end-users and contributors. I think it's currently only really used by GHC devs.

Check out https://github.com/haskellfoundation/stability/pull/2, it sounds like what you are asking for.

bergmark commented 2 years ago

I haven't seen it mentioned, but this reminds me of this organisation - https://github.com/haskell-pkg-janitors - which I guess had a similar aim; but ended up being quite inactive.

Thanks @silky, I had forgotten about haskell-pkg-janitors. I would hope that the difference in this proposal is that 1) every project has their own set of maintainers, which makes sure that if one package gets abandoned then other packages won't be affected. 2) We will have a rotation of people managing the project as a whole to keep the lights on.

That said, there is of course a risk that this project will become inactive as well, but I would argue that while that would be sad for me personally, I don't think it would hurt the community.

I'm also not sure if haskell-pkg-janitors is a failure? They don't host many packages, but there has been some recent activity. Perhaps there was a short spike of activity? That also sounds like a success. Likewise here, if after a few years we only host 5 packages, I don't think we necessarily failed. If this project becomes popular then that is great, if not, then we can learn from it.

bergmark commented 2 years ago

This proposal looks the closest to how I see the solution to delayed updates of packages in the Haskell ecosystem. And I think the proposal can be simplified (in my vision) to the following:

:3

Create a shared Hackage account HaskellStabilityGroup

I want to clarify that this is not work that the stability working group will be responsible for. The idea has spawned from discussions we've had, and I'm using this as a forum to gather feedback on this project.

Maintain a list of people with access to this account When a maintainer wants to provide stability guarantees to their users, they can do this by contacting the stability group, adding the HaskellStabilityGroup Hackage account to their package and grant write access to at least two people in the group That's it. There're some details that need to be specified (e.g. activity expectations from maintainers, how to maintain the list, etc.). But in my vision, the main reason behind the inability to update packages quickly is that all packages are managed by volunteers in their free time.

They don't have to upgrade their packages quickly and nothing can force them to do this.

But if they add other people to their projects and grant them merging and Hackage uploading rights, package upgrades can happen without the need to wait for the maintainer. A dozen of volunteers can migrate the entire ecosystem to a new GHC within a couple of weeks.

I am getting mixed feedback on this, some are wary of losing control of their projects, and some just want to make sure that all of the maintenance work gets done. Likewise, when hackage revisions were introduced some said "YOU'RE DOING WHAT TO MY PACKAGE?" while others though the hackage trustees should have the ability to do much more than just adjusting version bounds.

I don't think we need to pick one or the other here. If you join Haskell Party with your project you should be able to specify what you want. Can anyone who volunteers patch your library? What kind of changes are OK for these volunteers to do?

I don't think a separate organization for this is required. When the maintainer is MIA, someone from the stability group can fork their repo (to their personal GitHub account or even better to the haskell org) and update maintainership information.

I would like to avoid forking as I think it is confusing. What is the source of "truth"? How do we make sure changes get back to the source? Where should issues be posted? It also splits the maintainership, now there is a separate repository where the original maintainer may not have access. It is unfortunate that a github repository hosted under an individual's account cannot have multiple owners, the haskell-party github organization is mostly a workaround for this.

That said, we could definitely provide "we'll update the source if we have access, but otherwise fork" as an option.

Do you see downsides with transferring repositories?

Thank you for getting involved!

Ericson2314 commented 2 years ago

I am having trouble understanding how this is different from github.com/haskell. Maybe a comparison of those two as part of the proposal would be useful? Is this something that has to be different/separate, or would you be happy if github.com/haskell was just reformed along some lines?

chshersh commented 2 years ago

@bergmark

I want to clarify that this is not work that the stability working group will be responsible for

I'm okay if some other group of volunteers will do this. The stability group can discuss the proposal and, as usual, call for volunteers 🙂 I've just chosen the Hackage user name to reflect that it's related to the stability aspect of the maintenace.

Likewise, when hackage revisions were introduced some said "YOU'RE DOING WHAT TO MY PACKAGE?" while others though the hackage trustees should have the ability to do much more than just adjusting version bounds.

I think this is more of a social issue. Better visibility about Hackage trustee powers can help. For example, when you create an account on Hackage, you'll be presented a list of things to consider such as Hackage trustees can change dependencies of packages, etc. Some short and explicit "Terms and conditions" 🧑‍⚖️ Improving the situation around revisions would be nice as well but I believe this is not in the scope for this proposal.

The main idea here is that the process is opt-in. So people agree to have patches submitted and merged to their libraries without their review if they take long times to respond.

It is unfortunate that a github repository hosted under an individual's account cannot have multiple owners, the haskell-party github organization is mostly a workaround for this

Yes, I agree, it's unfortunate. It would be nice if there was an ability to transfer the repository under some organization after some time. Anyway, since other maintainers have write access to the repo, they can update the disclaimer saying that this project now lives "here" and ask people to contribute patches and issues to the other place.

Do you see downsides with transferring repositories?

The downside is that people might want to keep full control of their packages (being able to leave the stability support initiative, transfer the repository somewhere else, etc.). Also, having their name in the project helps with visibility for them.

Otherwise, if we ask people to give rights to their packages and transfer their repository somewhere, it may feel like the project doesn't really belong to them anymore.

@Ericson2314

Is this something that has to be different/separate, or would you be happy if github.com/haskell was just reformed along some lines?

My personal view is that I'd love all well-maintained packages to be in the github/haskell org but at this point in time, the purpose and scope of this org is not yet clear:

sjakobi commented 2 years ago

I think I'd like to be involved in this party as a maintainer. As a Hackage trustee I already spend time on addressing issues with other people's packages. With this organization, trustee work may become more effective, because it should make it easier to release compatibility fixes to Hackage.

I'm currently aware of two packages that I would like to bring into the fold of the new organization:

One concern I have about this project though is that some people may try to add packages that don't have a significant value to the Haskell ecosystem. Some packages never gain a significant user base or stop being relevant for one reason or another. As a maintainer in the new organization, I wouldn't want to spend attention and time on such packages.

I think it may be helpful if package additions would require some kind of justification. E.g.

This package has X direct reverse dependencies on Hackage, and Y additional transitive dependencies, of which Z are actively maintained. This package is also used in non-Hackage projects A and B. Company C uses it too.

Maybe this could be followed by a vote, where if, say, two maintainers say "I want to maintain this", the package is accepted.

(CCing @andreasabel who is currently maintaining a lot of previously under-maintained packages and who might have some thoughts on this proposal.)

hasufell commented 2 years ago

I'm against gatekeeping of any kind, so let's not make this into some form of stackage.

If there is a package that's mostly irrelevant to the community, but the author looks for help and there's people willing to help, then it should be supported regardless of its usefulness.

The main value I'd see in a proposal like this is less of an organized task force, but more of a platform where authors looking for help and volunteers looking for work are connected.

bergmark commented 2 years ago

\ I am having trouble understanding how this is different from github.com/haskell. Maybe a comparison of those two as part of the proposal would be useful? Is this something that has to be different/separate, or would you be happy if github.com/haskell was just reformed along some lines?

@Ericson2314: Sure, I'll add some mentions of github.com/haskell and other organizations. I don't think a repository living under github.com/haskell means anything for maintainers as it stands, but the organization is managed by haskell.org (I think!). For haskell-party to manage packages within github.com/haskell we would need wide organization permissions that may also give us access to other packages that live there, which I don't want us to have unless everyone in that org agrees. It seems simpler to start something new.

\ The downside is that people might want to keep full control of their packages (being able to leave the stability support initiative, transfer the repository somewhere else, etc.). A Otherwise, if we ask people to give rights to their packages and transfer their repository somewhere, it may feel like the project doesn't really belong to them anymore.

@chshersh: I don't think moving to the haskell-party organization prevents anyone from leaving. https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-organization says "If you have owner permissions in an organization or admin permissions to one of its repositories". The original maintainers would be repository admins.

But I can see that there could still be reluctance due to this, so perhaps it would be sufficient for us to say "We prefer if the repository is transferred because of \. If it is not transferred then we may do \"

\ I think I'd like to be involved in this party as a maintainer.

@sjakobi: Great, thank you!

With this organization, trustee work may become more effective, because it should make it easier to release compatibility fixes to Hackage.

Yes I would hope so too. Though the main benefit I see here is the explicit opt-in for you to do this work. While I was active in trustee maintenance the most stressful thing for me was the worry of stepping on people's toes.

\ I think it may be helpful if package additions would require some kind of justification. \ I'm against gatekeeping of any kind, so let's not make this into some form of stackage.

@sjakobi, @hasufell: I agree with both of you here... I think it would be good to have some guidelines stating that a project should be of some "importance", but I would not want us to have to review applications and decide which packages are worthy. I think we can start out very lenient here and see how things progress.

@sjakobi Perhaps this is coming from you volunteering to help out and not wanting to be spread too thin on things that might be unimportant? The way I envision this is not that you sign up to maintain every package, you would instead choose which ones you care about. So you are free to have your own criteria be as strict as you please. If we announce the need for maintainers of a package but no-one volunteers, then we can clearly mark the package as in need of maintainers rather than kicking it out of the organization. We also need to be able to manage projects that may be important today, but be replaced with something else down the line.

Ericson2314 commented 2 years ago

@bergmark

@Ericson2314: Sure, I'll add some mentions of github.com/haskell and other organizations. I don't think a repository living under github.com/haskell means anything for maintainers as it stands, but the organization is managed by haskell.org (I think!). For haskell-party to manage packages within github.com/haskell we would need wide organization permissions that may also give us access to other packages that live there, which I don't want us to have unless everyone in that org agrees. It seems simpler to start something new.

OK that helps I would definitely include. GitHub does make managing permissions extremely annoying and manual, but the tradeoff is that having multiple orgs with similar packages is confusing for users of packages, and also for coordinating on the HF side.

IMO what you are actually trying to do is promote collective stewardship of packages, which is good. But by phrasing this as "let's make yet another org" rather than "let's think about github.com/haskell and CLC charter", you might avoid some political hurdles up front but the result is more balkanization and confusion --- just the sort of thing the centralization in the form of the HF was supposed to prevent.

Ericson2314 commented 2 years ago

Github does have "teams", so you could within github.com/haskell have different teams for different packages based on a trade off "very important, must manage carefully"-------"less important, good that more people can step in an keep things moving".

Having a unified organizational structure, but many such "tiers" for flexible approach strikes me as the ideal.

tomjaguarpaw commented 2 years ago

Relevant Twitter thread: https://nitter.net/GabriellaG439/status/1523432665541537793

sjakobi commented 2 years ago
I think it may be helpful if package additions would require some kind of justification. I'm against gatekeeping of any kind, so let's not make this into some form of stackage.

@sjakobi, @hasufell: I agree with both of you here... I think it would be good to have some guidelines stating that a project should be of some "importance", but I would not want us to have to review applications and decide which packages are worthy. I think we can start out very lenient here and see how things progress.

@sjakobi Perhaps this is coming from you volunteering to help out and not wanting to be spread too thin on things that might be unimportant? The way I envision this is not that you sign up to maintain every package, you would instead choose which ones you care about. So you are free to have your own criteria be as strict as you please. If we announce the need for maintainers of a package but no-one volunteers, then we can clearly mark the package as in need of maintainers rather than kicking it out of the organization. We also need to be able to manage projects that may be important today, but be replaced with something else down the line.

Alright, that seems reasonable to me. If I don't want to be bothered about certain packages, I can simply turn off notifications for those.

In the meantime I keep stumbling across more packages that seem to be waiting for maintainers to take care of them:

I doubt that there will be any shortage of work for this organization! ;)

bergmark commented 2 years ago

Thanks again for all of the feedback! I have gotten lots of good input here and via other channels. I plan to rework the text a bit. The core idea is pretty much the same, but I notably did not do a good job in explaining what this project means for participants.

jamesdbrock commented 1 year ago

I love this proposal in its current form. Let’s launch this. We can tweak the explanation later. I want to help, and I want to add some packages and recruit a bunch of Stewards. DM me on Haskell Discourse @bergmark .

liamzee commented 1 year ago

Haskell Github Trust is already active, and they've indicated their desire to merge into any HF approved organizations on Reddit.

===

I'd also suggest we upgrade Hackage for this purpose; i.e, allow groups and assigning group access levels so that in an ideal case, we don't end up with 100 people with full access to everything being held by Haskell Party / Haskell Github Trust.