search

haskellfoundation / tech-proposals

The Haskell Foundation Tech Proposal Process
Other
69 stars 29 forks source link

GHC's base libraries: Combining stability with innovation #51

Closed Ericson2314 closed 1 year ago

Ericson2314 commented 1 year ago

This document describes an plan agreed between the GHC Team and the Core Libraries Committee, saying how we plan to work in partnership to reconcile the goals of innovation and stability for GHC and its ecosystem.

Formally, then, it is not so much a proposal as a record of an outcome. We are, nevertheless, using the Haskell Foundation Technical Working Group proposals repo, so that we can have a permanent public record of how we plan to work, so that others can comment, and so that the document can be polished to add clarity where necessary.

Rendered

Bodigrim commented 1 year ago

Dear CLC members, following our earlier private discussion, I understand that we are in agreement with the proposal in principle. This PR is a good opportunity to polish wording, clarify statements and generally take one more final look. However, let's not scope creep, unless absolutely necessary.

@tomjaguarpaw @chshersh @hasufell @mixphix @parsonsmatt @angerman

hasufell commented 1 year ago

I rather like @nomeata's suggestion that Hackage reject packages with explicit ghc-internals dependencies, it seems like the right abstraction layer to do so: if someone really, really wants to depend on it, they can still do so privately.

I disagree.

ghc-internals is internal API, but properly PVP versioned. Power users writing high performance code or alternative Preludes may have very legitimate use cases of importing it. Rejecting those packages on hackage is wrong.

nomeata commented 1 year ago

Power users writing high performance code or alternative Preludes may have very legitimate use cases of importing it.

That sounds very reasonable, but very different from “discourage by all means possible”, and probably deserves clarification. Which one is it?

hasufell commented 1 year ago

Power users writing high performance code or alternative Preludes may have very legitimate use cases of importing it.

That sounds very reasonable, but very different from “discourage by all means possible”, and probably deserves clarification. Which one is it?

"Discourage" is the correct wording, it does not imply "prohibit".

angerman commented 1 year ago

:+1: This seems like a step in the right direction. I hope that this is the first step of decoupling GHC and GHC releases from library upgrades rippling through the whole ecosystem. 🤞

hasufell commented 1 year ago

The two goals are in risk of conflict, and we have seen such conflict in the past.

Feels like it unnecessarily pits the CLC and GHC team against each other. As someone who has contributed to GHC, and had whose primary interest is stability, and proper system integration, it feels unnecessarily discouraging. We do want people to contribute to GHC, that care about stability.

I strongly disagree that we want to have a CLC (stability stalwarts) vs. GHC (innovators) view set out here. GHC needs to welcome people who care about stability and backwards compatibility and the CLC also needs to welcome advocates for innovation.

This is why I had proposed a different wording that doesn't focus on exact goals, but on the autonomy of GHC and CLC, which makes sense for its own sake:

  1. CLC has full autonomy over decisions affecting 'base' (API, performance, semantics), including its dependencies.
  2. GHC developers seek more autonomy over GHC internals and experimental features.

We want CLC and GHC to be able to exercise different goals (e.g. stability) and workflows (release management/versioning), but be sure that CLC's decision-making is not compromised, e.g. by changes that indirectly affect 'base'. We also want to be efficient and avoid proposals that are not strictly relevant to proper 'base' API.

@simonpj was unhappy with "including its dependencies", which I still don't understand, because it's absolutely true.

So let me propose this alternative wording again. Let's not focus on exact goals, because they are rather hard to express when two heterogeneous groups are involved. Let's focus on autonomy and how collaboration is supposed to work.

chreekat commented 1 year ago

A quick comment on the question of hiding ghc-internals on Hackage. My experience searching for information that has been "helpfully" hidden from me leads me to recommend against. Much better would be to build a standard way to mark a library as "internal" that tooling can use to put big honking warnings around. More information rather than less information. Plus it would allow other library authors to use the same mechanism, ameliorating a general problem and playing well with the support for multiple public libraries that's shaping up.

A caveat: for now, the "big honking warnings" should just go in the Haddocks, to decouple this effort from any hypothetical tooling changes.

simonpj commented 1 year ago

So let me propose this alternative wording again. Let's not focus on exact goals, because they are rather hard to express when two heterogeneous groups are involved. Let's focus on autonomy and how collaboration is supposed to work.

Indeed, the wording in the document is very close indeed to what you suggest (not a coincidence) including "including its dependencies". (I elaborated a bit what the GHC team seeks.) Perhaps there are other changes you would like to see?

GHC needs to welcome people who care about stability and backwards compatibility and the CLC also needs to welcome advocates for innovation.

I strongly agree with this sentiment @angerman! Indeed the doc says "Each of us places different emphasis on these goals, but respects them both. The CLC does not want to constrain GHC unnecessarily; and the GHC team does not want to undermine CLC's efforts, indeed quite the reverse!". So I think we all agree about what we want to say; the only question is, how to say it. Do you have a concrete suggestion for how we might say this better? Clearly it is not coming over as intended, to you anyway. Let's fix that!

angerman commented 1 year ago

@simonpj I tried rewording it a few times. It's hard. Harder than I thought it would be. I actually like @hasufell's wording a lot. The current wording in the proposal seems to lead too strong with "The CLC" vs "The GHC Team", and assigning them different roles from the the get-go in the itemisation. The current phrasing

  • The Core Libraries Committee seeks to exercise its control over decisions affecting 'base' (API, performance, semantics), including its dependencies.
  • The GHC team seeks the freedom to

make it sound incompatible from the onset. Which I don't think it is, nor needs to be.

Contrary to that I feel that @hasufell's wording is much more neutral:

  • CLC has full autonomy over decisions affecting 'base' (API, performance, semantics), including its dependencies.
  • GHC developers seek more autonomy over GHC internals and experimental features.

While I find

We want CLC and GHC to be able to exercise different goals (e.g. stability) and workflows (release management/versioning), but be sure that CLC's decision-making is not compromised, e.g. by changes that indirectly affect 'base'. We also want to be efficient and avoid proposals that are not strictly relevant to proper 'base' API.

to come across bit one-sided. We should probably add:

At the same time we do not want to constrain the development of GHC itself.

Ultimately I see the base split as

-----------------------.
                       |
ghc - ghc-internals - base - <wider ecosystem>
                       |
-----------------------'

base as well as the other libraries mostly under supervision of the CLC are the public interface of the standard library exposed to the community. And we want to ensure that the wider ecosystem can seamlessly depend on a stable interface, while GHC has a bit more freedom to change (and the acknowledgement that base is a bit too fat for this right now).

If we can keep the public interface stable, we can trivially keep compatibility across a few versions, which to me sounds like one of the goal of this proposal. Maybe I'm mistaken?

simonpj commented 1 year ago

@simonpj I tried rewording it a few times. It's hard. Harder than I thought it would be. I actually like @hasufell's wording a lot. The current wording in the proposal seems to lead too strong with "The CLC" vs "The GHC Team", and assigning them different roles from the the get-go in the itemisation.

Thanks. I'm not getting the "setting up conflict" vibes, but I'm entirely happy to change the words, since I think we all agree on the intent. How about this?

This proposal allows the GHC team and the Core Libraries Committee to work in a productive partnership. What we want to achieve is this:

The proposal sets up mechanisms (e.g. what packages exist), responsibilities (e.g. who curates which package), and processes (e.g. who should be consulted and when) that support both CLC and the GHC team to follow their respective goals without tripping over each other.

The proposal is based on https://github.com/haskellfoundation/tech-proposals/pull/47, but is independent of it. You can find more background in https://github.com/haskell/core-libraries-committee/issues/146.

michaelpj commented 1 year ago

I find the lifecycle of and responsibility for ghc-experimental a bit weird.

Here's an alternative proposal:

I think this would have some advantages:

The downside is that the CLC has to get involved earlier. But I think the current proposal is false economy: since the CLC will have to get involved later anyway, there doesn't seem to me to be much advantage in delaying it.

tomjaguarpaw commented 1 year ago

As a contributor, if the CLC must eventually approve my changes, I would very much like them to weigh in early before I do lots of work!

I don't understand this objection. If you want something in GHC then you propose it to Team GHC and, if successful, you and/or they add it, and parts of it may be exposed through ghc-experimental. You're then more than welcome to expose a stable interface to your work through your own package. It really has nothing to do with base at all!

michaelpj commented 1 year ago

If you want something in GHC then you propose it to Team GHC and, if successful, you and/or they add it, and parts of it may be exposed through ghc-experimental. You're then more than welcome to expose a stable interface to your work through your own package.

Most GHC proposals that I've seen which require newly exposed functions are pretty clearly aiming at getting them into base. Proposing that they might never go into base and instead be exposed through some other library is certainly possible but seems like a very different workflow! (And then ghc-experimental seems like a bad name for the library...). Consider e.g. https://github.com/ghc-proposals/ghc-proposals/pull/330. Ignoring the changes to existing parts of base (which clearly need CLC approval), it also proposes adding quite a lot of things. It seems like a pretty bad process failure if those could get stuck in ghc-experimental because nobody asked the CLC when the proposal got accepted.

If I understand you rightly, you would say "This is fine, in such a case the author can just distribute that code via another package"?

tomjaguarpaw commented 1 year ago

If I understand you rightly, you would say "This is fine, in such a case the author can just distribute that code via another package"?

Yes, absolutely. If and when appropriate (based on usage numbers, stability, and other concerns) the functionality can be absorbed into base. This has been my expectation of how the ghc-experimental flow could work since I first heard of it.

michaelpj commented 1 year ago

Can we add some examples of the various workflows to the proposal, then? I think that would help a lot.

simonpj commented 1 year ago

Can we add some examples of the various workflows to the proposal, then? I think that would help a lot.

If anyone would like to offer to some concrete text, that would be great. We could have a section at the end on "possible workflows".

I'm listening to @bodigrim who counsels against scope creeep. But a clearly-seprarated discussion/illustration section, that doesn't form part of the proposal but has dicussion about how it might work in practice, would be fine I think. It just needs someone to write it!

hasufell commented 1 year ago

I find the lifecycle of and responsibility for ghc-experimental a bit weird.

  • It is a staging ground for things going into base

No, it is not.

ghc-experimental is not under CLC purview and not maintained by CLC. It's entirely up to GHC HQ/SC how to use it.

CLC doesn't necessarily care about it at all. But it's a possible mechanism for GHC to:

Proposals may still be rejected. We have full autonomy.

GHC SC is welcome (and encouraged) to source CLC's opinion before accepting GHC proposals that possibly touch base, but it's not a requirement.

But if you don't, you can't complain later that your proposal as a whole failed due to CLC rejecting it.

silky commented 1 year ago

Meta question: Is there a picture anywhere that explains the different GHC "groups" and who/what they care about?

Feels like someone could do a great service to the community just by making that picture ... (it would certainly help me!)

tomjaguarpaw commented 1 year ago

Is there a picture anywhere that explains the different GHC "groups"

When you say "groups" what do you mean? Do you mean bodies like the core libraries committee, GHC steering committee, etc.?

silky commented 1 year ago

When you say "groups" what do you mean? Do you mean bodies like the core libraries committee, GHC steering committee, etc.?

Yes; it seems to me a lot of time in this thread is being spent on the various domains/interests/roles not being clear? (Could just be my misunderstanding.)

hasufell commented 1 year ago

When you say "groups" what do you mean? Do you mean bodies like the core libraries committee, GHC steering committee, etc.?

Yes; it seems to me a lot of time in this thread is being spent on the various domains/interests/roles not being clear? (Could just be my misunderstanding.)

Committees are documented here: https://www.haskell.org/community/

Efforts to document more of the hierarchy are discussed here: https://github.com/haskell-infra/www.haskell.org/issues/60

Please comment there.

silky commented 1 year ago

@hasufell

Committees are documented here: https://www.haskell.org/community/

No; that's not a complete list; but thanks, I didn't know about that other issue.

tomjaguarpaw commented 1 year ago

that's not a complete list

Could you please open a new issue on the haskell.org repo explaining which are missing? I'll make sure they get added.

Bodigrim commented 1 year ago

I'm happy to announce that the proposal has been approved unanimously by CLC.

simonpj commented 1 year ago

Thank you Andrew!

Simon

On Tue, 11 Jul 2023 at 22:45, ˌbodʲɪˈɡrʲim @.***> wrote:

I'm happy to announce that the proposal https://github.com/haskellfoundation/tech-proposals/pull/51/commits/c157f56c4f77ee1c7caaf5c1433296427971ed94 has been approved unanimously https://groups.google.com/g/haskell-core-libraries/c/10SyhA9Lm98/m/Zimb2ZZUAAAJ by CLC.

— Reply to this email directly, view it on GitHub https://github.com/haskellfoundation/tech-proposals/pull/51#issuecomment-1631552317, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEOY62ZSZ6RZ4LTLPSKSOLXPXCQ5ANCNFSM6AAAAAAZUUZN5E . You are receiving this because you were mentioned.Message ID: @.***>

hasufell commented 1 year ago

I'm happy to announce that the proposal has been approved unanimously by CLC.

Sorry to nitpick, but I don't see the vote from @mixphix on the mailing list. Did they reply privately?

mixphix commented 1 year ago

Oh, my apologies, I did indeed reply privately by mistake.

hasufell commented 1 year ago

I think it is time to merge @Ericson2314 or does the HF want to do anything before that @david-christiansen ?

simonpj commented 1 year ago

I think it is time to merge @Ericson2314 or does the HF want to do anything before that @david-christiansen ?

I agree.

Relatedly, once we have done so, I'd like to start a conversation about naming conventions for modules. See this thread. Where would be a good place to have that conversation? As an issue (or a proposal) on this HF github? On the GHC github? Where?

david-christiansen commented 1 year ago

For the sake of formalities, TWG will have a vote before merging, but I'm pretty sure it won't be controversial.

As far as that other conversation goes, I have some ideas - I'll get back to you soon.

david-christiansen commented 1 year ago

The TWG agrees that this proposal has been approved by the relevant parties and is useful - we unanimously voted to approve.

david-christiansen commented 1 year ago

Thank you all for this process. I think it led to a good outcome!