search

haskellfoundation / tech-proposals

The Haskell Foundation Tech Proposal Process
Other
69 stars 29 forks source link

Towards a better end-user experience in tooling #48

Closed hasufell closed 5 months ago

hasufell commented 1 year ago

Most of the points in this document were firstly discussed with @bgamari in a meeting. So this is basically a refinement of those with additional context.

Some actions have already been taken, so we want to use this thread to track progress and also inform the public.

Although the title is very broad, we focus on release and bindist issues here, because those are the most impactful at the moment.

Context

The tooling ecosystem today consists of at least the following major components:

  1. GHC
  2. Cabal
  3. Stack
  4. HLS
  5. GHCup
  6. VSCode Haskell extension

The end-user experience depends on how well these tools interact and synchronize releases/updates.

Although the tooling ecosystem has improved, we frequently get user reports about issues that require deep knowledge about existings bugs, known workarounds or how these tools actually interact, breaking the plug-and-play experience.

Problem statement

Most of the tools mentioned above have different maintainer teams, with some overlaps and individuals that regularly contribute across them.

We present here a dependency table between those tools:

GHC Cabal Stack HLS GHCup VSCode Haskell
GHC - (hadrian) - - (CI) -
Cabal X - - - (GHC installation) -
Stack X - - - (GHC installation) -
HLS X X X - X -
GHCup X X X X - -
VSCode Haskell X X X X X -

This gives a superficial overview of how intertwined the tools are.

The end user often doesn't know these details if they follow the installation instructions of the VSCode haskell extension or GHCup.

Frequently raised issues include:

  1. GHC does not work on their system
    • missing bindists for minor platforms
    • issues with darwin M1 or windows
    • GHCup or stack install incompatible bindists on their system
  2. HLS does not work on their system
    • missing binaries for a particular GHC version
    • GHCup installs incompatible bindists on their system
  3. Releases are out of sync
    • GHCup or stack lag behind GHC releases
    • HLS lags considerably behind GHC releases

The reasons for these issues are multi-faceted. Some of them are not of technical nature.

I want to focus on a couple of interactions only and provide evidence of problems:

  1. GHC and the downstream consumers GHCup and stack that consume the bindists
  2. HLS and all other tools, because it's the most challenging to release

GHC bindists

GHC bindists are consumed mainly by GHCup and by stack:

Both these tools provide means of installing GHC versions by providing a unique mapping. GHCup uses a distro+distroversion mapping, while stack uses platform+library-sonames as index. Both have their own caveats.

When new releases are announced, the GHCup and stack maintainers investigate the available bindists GHC has provided with the release and decide on how to update their mappings, so that users of any platform and linux distro get a working bindist.

The main issue here is that neither stack, nor GHCup maintainers have direct influence on the decisions of what bindists are created and how, although they have high stakes in it. They are not consulted or informed consistently (although occasionally) on every change and are not required to sign off on any non-trivial changes.

Rather, they are more downstream consumers that, most of the time, have to deal with the end result post-release. There are rarely testing or RC phases for bindists and GHC maintainers are reluctant to update bindists in case they have issues (this happened only once when profiling libs were missing).

As a consequence, GHCup maintainers have had to invest considerable amount of work into curating GHC bindists. These efforts can be seen here: https://downloads.haskell.org/ghcup/unofficial-bindists/ghc

The reasons so far included:

The most recent disruption was an update of the Fedora bindist, which was regarded by both GHCup and stack maintainers as the linux bindist with the highest portability. When the CI configuration was changed to use Fedora 33 instead of 27 to build the binaries, the portability assumption broke due to a too recent GLIBC version and no testing was able to catch it.

It is still causing disruption today:

GHC releases

GHC releases have become more frequent in recent times and have increased pressure on tooling to keep up and support these new versions.

GHC release process until recently didn't include many downstream consumers and there was no explicit communication. Some release candidates existed only for major versions.

Tooling maintainers would have to follow the issue tracker and the mailing list closely to be up to date with issues that are relevant to them.

As such, releases tended to overwhelm not only GHCup, but especially HLS maintainers, where supporting new GHC versions is non-trivial.

HLS releases

Releasing HLS is non-trivial and includes:

  1. updating the codebase to support a new GHC version if one was just released
  2. fixing plugins, pinging their maintainers or disabling them when they stopped working
  3. dealing with the complex release CI, which takes many hours to run

In terms of dependencies, the cycle of releasing a new HLS version may look like this:

  1. new GHC version gets released
  2. HLS needs to wait for GHCup to support the new version, because GHCup is needed to build the binaries for the correct platforms (more on ABI compatibility later)
  3. HLS can start its release process, deal with CI, then publish a release
  4. now GHCup metadata needs to be updated as well, for the new HLS version to land

Issues with GHC bindists will bubble up to HLS and impact the end-user experience of someone who just installed the Haskell VSCode extension and doesn't know anything about the tooling (see the GLIBC issues described above).

So far, this pipeline is quite separated with no tight coordination. As such, support for GHC 9.2.5 took a long time to finally land in HLS 1.9.0.0, which has been impacting users as well.

One major reason these things became more complicated is that HLS had to switch to dynamically linked bindists in order to avoid issues with the internal linker and TH, see:

Ways to improve

The main goal of the suggested improvements is that the end user never knows what is going on behind the scenes to make the tools work smoothly with each other and have releases that are tightly in sync. The solutions are not self-serving and should be rejected or reconsidered if they don't bring us closer to this goal.

Binary distribution metadata

https://gitlab.haskell.org/ghc/ghc/-/issues/22686

GHC maintainers should maintain a metadata file that describes each bindist (in e.g. json or yaml) and includes at least the following information:

The idea behind this is that all changes to bindist are at least very strictly visible post-release and can be consumed by tools, linters or just diffed by a human to get an overview.

Synchronous release meetings

However, the previous point does not solve communication issues and does not prevent mistakes pre-release.

To achieve that, I propose to hold synchronous meetings 1-2 weeks prior to a release to discuss:

  1. changes that affect tooling maintainers (cabal, HLS, ghcup, stack), especially any changes to bindists or build system
  2. getting last input from tooling maintainers (e.g. outstanding bugs that are still affecting them, but are not regressions)
  3. sharing current state of most prominent issues that affect end-users (on any level of tooling) and evaluating whether the upcoming release can make a difference
  4. getting in-person updates about things that affect each other, without expecting everyone to follow every single bug tracker and mailing list of every other tool

This should make it possible to catch issues before a release and make maintainers aware of upcoming issues.

Making it feasible to change bindist mappings

A bindist mapping is a mapping of a platform (e.g. Ubuntu 20.04) to a binary distribution of a GHC version. At the moment, these are the problems when changing bindist mapping in ghcup after they've already been released (mutating them, so to speak):

  1. cabal indexes the store by ghc version only (not ABI) and the boot packages in the store metadata are generally not hashed. This can lead to miscompilations and has a large chance to break CIs. Cabal would either have to index the store by GHC ABI or somehow calculate hashes for boot packages as well. Also see https://github.com/haskell/cabal/issues/8114.
  2. Stack uses ghcup --info to index package databases, which is a little more robust, but also fallible.
  3. When a HLS version is already released, changing mappings requires a new HLS release as well, to avoid that it crashes on startup with "ABI mismatch" errors. Even if new binaries are out, users who don't do a fully consistent update may still face such issues.
  4. ghcup itself currently has no concept of "revisions" (that is: a new packaged version of the same tool version, e.g. GHC 9.4.4-r1). The reason is that this is hard in terms of usability and design space. Also see https://github.com/haskell/ghcup-hs/issues/361

@bgamari opines that the right solution to (1) is for boot packages to have proper ABI hashes computed by Hadrian. These could then be factored in to the unit ids of a package's dependencies. Some refactoring of GHC's internal tracking of interface hashes may be necessary for this (see https://gitlab.haskell.org/ghc/ghc/-/issues/20810). See also https://gitlab.haskell.org/ghc/ghc/-/issues/21590.

This may be a major effort that we need to find a contractor for.

It's the main reason why GHCup maintainers have to be so paranoid about bindists being correct and often leads to bugfixes having to wait for the next GHC release (as is the case right now... Fedora 27 bindists were added back to GHC release CI, but tooling maintainers were not aware and ghcup and stack metadata was already released).

Developing a holistic tooling release workflow

The current release workflow can benefit from more automation as well as more cross-checking between maintainers, e.g.:

  1. GHC automatically sends PRs to ghcup-metadata repo on releases
  2. significant ghcup-metadata changes (adding of new tool versions) are propagated in some way that can be subscribed to
  3. HLS automatically sends PRs to ghcup-metadata repo on releases
  4. there is a cross-maintainer sign-off policy, where
    • significant PRs in GHC that affect bindists or build system need approval by stack/ghcup/HLS maintainers
    • ghcup-metadata PRs need approval by the respective tool maintainer (GHC, stack, HLS, cabal)
    • HLS releases need final sign-off by HLS and ghcup maintainers before publish
  5. Document the workflow sufficiently and describe current decisions that are in effect (e.g. why dynamically linked HLS is used)

This would synergize with the synchronous GHC release meetings and may be expanded. E.g. currently, we have monthly HLS meetings and @mpickering is already attending them.

The idea behind this is to make the tool boundaries a little bit less strict and take shared responsibility and enable to discuss the end-user experience regularly and work towards shared solutions (relevant for the next section).

Creating a shared project for tooling

I suggest to set shared goals for the entire tooling suite that focuses on the end user and then create a project on github or an epic on gitlab to track the progress. The reasoning behind this is that a lot of issues (like GHC ABI) affect multiple or all tools and tracking progress, discussions etc. across multiple issue trackers is cumbersome and causes poor visibility of high priority issues.

Issues living in this project should naturally have high priority for all designated maintainer teams.

This could be combined with a bounty system backed by the Haskell Foundation.

Increasing the bus factor across critical tooling

Many of the mentioned issues exist, because all tools are short of maintainers and only some tools have paid contributors.

GHCup has only one maintainer, who is not paid. Not many people are able to do HLS releases and understand all the existing issues. Cabal is understaffed too.

If we develop no strategy on how to deal with the bus factors, all efforts to improve the situation may eventually regress when knowledge and engagement gets lost.

Open question: outsourcing of GHC bindist creation

There has been some discussion whether outsourcing of bindist creation (e.g. for esoteric linux distros or tier3 platforms) would be an option too.

Practically, this is already happening partly in GHCup, but the idea was that GHCup completely relies on its own bindists.

This is more of an open thought than an actual proposal at the moment.

Closing thoughts

As can be seen, not all issues here are technical and it has been my impression that we've focused too much on that part.

Automation and CI are all great. But they don't ultimately solve the coordination issue.

It can be debated whether we would actually need a proper release coordinator who works across tooling boundaries.

Right now, I'm not sure how that would look like. I propose to try the above improvements first and maybe some of the friction will already be gone.

There are also more topics to be covered, e.g. how to unify the GHC installation methods to cause less friction with HLS+stack. But I feel that is for another day.

hasufell commented 1 year ago

Pinging some maintainers, for whom this is relevant: @mpilgrem @michaelpj @fendor @mikolaj

mpilgrem commented 1 year ago

@hasufell, thanks for the ping.

From the perspective of Stack (the tool), a new GHC version is usually not a major event. The needs of other tools seem to be more demanding. For Stack, it involves updating: (a) https://github.com/commercialhaskell/stackage-content/blob/master/stack/global-hints.yaml for the package versions of the libraries that come with GHC on Windows and non-Windows; and (b) https://github.com/commercialhaskell/stackage-content/blob/master/stack/stack-setup-2.yaml for URLs, file sizes, SHA1 hashes and SHA256 hashes of GHC binary distributions. For me, (a) and (b) are manual processes. If nobody else has got there first, I hurry to do them when I read on the Haskell Community that a new GHC version has been released.

Step (b) is easy if the current structure is 'like' the most recent previous structure. It requires more thought if something 'structural' changes - like 'Fedora 27' disappearing and 'Fedora 33' appearing. If it is only me doing the thinking, then Linux-based structural changes can introduce a hiatus, as I don't have much Linux experience to draw upon.

Mikolaj commented 1 year ago

I agree closer coordination is very much needed. I'd add Haddock to the mix. I suppose Hackage is covered under cabal?

Ericson2314 commented 1 year ago

@bgamari opines that the right solution to (1) is for boot packages to have proper ABI hashes computed by Hadrian.

Absolutely, and in general I think most of the issues we have with our build process relate to tech debt that is possible to fix. Increased meetings / social coordination is always good, but in this case we should also simply invest in making the underlying technical situation less complicated. It's very feasible.

Kleidukos commented 1 year ago

I agree closer coordination is very much needed. I'd add Haddock to the mix. I suppose Hackage is covered under cabal?

@Mikolaj Fortunately (and unfortunately), the historical proximity between GHC and Haddock hasn't led to to major logistical problems. This may change in the near future but if we go on with the foundations described by Julian, I'm not worried.

abarbu commented 1 year ago

This is great! But it misses nix. According to the latest survey about 35% of users are relying on nix. https://taylor.fausak.me/2022/11/18/haskell-survey-results/

Haskell is, as far as I know, the only language that has two totally separate nix ecosystems. Worse, they're mutually incompatible. One is better at distributing software, the other is better at building and developing it.

Documentation is a mess with posts confusing them. They have completely different user-facing APIs.

  1. It would be nice to also coordinate releases with haskell.nix, as right now new compilers slowly trickle in with significant bugs. For example, 9.4 is still partially broken without workarounds. https://github.com/input-output-hk/haskell.nix/issues/1809

  2. We should strive to have only one infrastructure for Haskell in Nix. What's the point of wasting all of this effort to just end up with two sets of bugs, two sets of inadequate documentation, and a very confusing nix setup? And why would we burden users with having to learn two completely different APIs?

Kleidukos commented 1 year ago

No it really does make sense that nix is not included in this very specific proposal. In its current incarnation, with the current workforce at our disposal and the current repartition of expertise in the community, nix is a burden to maintain. I mean no offence by this, nor do I aim to trigger a flamewar, but it is true.

We should strive to have only one infrastructure for Haskell in Nix. What's the point of wasting all of this effort to just end up with two sets of bugs, two sets of inadequate documentation, and a very confusing nix setup? And why would we burden users with having to learn two completely different APIs?

Seems like these questions would be best addressed to IOG and the other players in the space. Until there is a massive effort to bring Nix to the masses, outside of the club of passionate people who are breaking their back maintaining the infrastructure, we can't do much but applause the principles of Nix, and shedding a tear for our martyrs.

angerman commented 1 year ago

@abarbu I appreciate you bringing this up. I'm however afraid we will not see a unification of Nixpkgs haskell infra and haskell.nix, they are philosophically very different, and also server almost orthogonal goals. Hence I believe (2) is simply not attainable. Haskell.nix will focus on versatility and usability for haskell projects, and to that extend gladly pay any IFD or other penalty, if it means the user has to deal with nix less; it also means we try to support a larger set of compilers, as projects can depend on older ones that were dropped from Nixpkgs. On the other hand Nixpkgs primary haskell infrastructure is mostly based on a stackage snapshot, and the primary goal is to support haskell packages in Nixpkgs, and secondarily the development of haskell packages. Therefor depending on a stackage snapshot to increase sharing makes a lot of sense, but that's not what your custom haskell project needs. However, I'd like to emphasise that haskell.nix can also be used for distribution, it's just much worse at distributing lots of Haskell packages with lots of dependency sharing between those packages. haskell.nix relies on cabal for dependency resolution--so haskell.nix stays as close to your cabal build invocation as possible--and therefore every project will get it's own build plan, with limited sharing (unless you use a stackage like freeze file).

The issue you bring up in (1) is fundamentally the issue of lib:ghc not being re-installable. Haskell.nix could probably improve here a bit. For any build plan you have, if you depend on lib:ghc, you'll also depend on the transitive closure of lib:ghc, and becuse lib:ghc is non-reinstallable, that transitive closure isn't re-installable either. This included array, directory, ...; haskell.nix tries to make as much re-installable as possible out of the gate, but if you end up trying to re-install lib:ghc this currently fails (and isn't really that safe anyway).

Now why does adding new compilers to haskell.nix take so long? For 9.4 upstream just broke it in many ways. We now have Hadrian, which fails to properly work for cross compilers. Upstream also decided to move iserv-remote out of tree, and rename parts of it; which ultimately broke a lot of haskell.nix code. I'm not going to go any further into this, as it would just end up in a massive rant.

@Kleidukos I tend to agree that we should not focus too much on nix. I will however point out that our nix infrastructure is currently the only one that can sensibly exercise ghc as a cross compiler. I'm not sure where haskell_rules is right now though.

haskell.nix is a community project, it is currently hosted in the IOG namespace, so maybe we should consider moving it somewhere else, if that would make it more obvious that it's a community project. There is one person who's primary job the maintenance of haskell.nix is, and has been consumed mostly with the fallout of upstream breakage (and ensuring all our cross targets still work).

Kleidukos commented 1 year ago

@Kleidukos I tend to agree that we should not focus too much on nix. I will however point out that our nix infrastructure is currently the only one that can sensibly exercise ghc as a cross compiler.

@angerman Just for the record, I fully agree with your statement.

Ericson2314 commented 1 year ago

The issue you bring up in (1) is fundamentally the issue of lib:ghc not being re-installable.

Focusing on making everything reinstallable is probably the number one thing that would benefit Nix, but it also is a high contender for the best thing to benefit everything else. So we can focus on that, not worrying about Nix in particular, and yet not neglecting Nix either.

jberryman commented 1 year ago

fundamentally the issue of lib:ghc not being re-installable

Isn't this already done? https://gitlab.haskell.org/ghc/ghc/-/issues/19896

dark-ether commented 1 year ago

firstly i will have to say i am biased as i use nix for my haskell projects. however if this issue is to be focused on the end user we shouldn't ignore the 35% that use nix + possibly something else , in my personal experience using nix has been better than using ghcup but i previosuly were on arch linux so your mileage may vary.

i do believe that including the nixpkgs haskell maintainers in these discussions would help them.

also nix tests more packages than stackage for instance i went to nixpkgs hydra jobsets and then searched for haskell and found that 6 days ago they built the ABList package for x86_64-linux and aarch64-linux this package isn't on stackage and i have never heard of it before, also they maintain a ghcHEAD version of ghc which is ocasionally updated. actually there is a pr to update it right now.

what i want to say is that nix should have importance close to stack in this discussion, not the main point but certainly not ignored, if we se the survey it says that 33% use nix and 12% use haskell.nix.

@Kleidukos i don't know if you use stack but to anyone that does surely if someone said that including stack was a maintenance burden and implied that it should be completely ignored you would at least object right?

that said if the situation is as bad as said maybe we should focus only on cabal and hls which are absolutely essential? if i am not mistaken both nix and stack internally use cabal and it is significantly harder to develop without lsp.

simonmichael commented 1 year ago

Great discussion; just to interject 2c, I don't believe 35% of Haskell users or would-be Haskell users use nix, I think that's partly sampling bias.

angerman commented 1 year ago

fundamentally the issue of lib:ghc not being re-installable

Isn't this already done? https://gitlab.haskell.org/ghc/ghc/-/issues/19896

Thanks for digging this up, so this was actually merged 🤔

hasufell commented 1 year ago

So, the reason I did not bring up nix is mainly because I have no interest in it and will not work on it.

This issue was mainly raised because of the problems GHCup and HLS have with bindists and release coordination. Nix does not use bindists.

If nix maintainers will benefit too from joining synchronous release meetings, then they surely will be included.

I'd like to focus on the mentioned issues wrt bindists and release coordination, though. Yes, the thread title is a bit broader and yes it's important to collect more topics that need attention.

I think @angerman has sufficient visibility about what nix and their maintainers need from GHC team, so I guess this document can serve as a template for demanding improvements in a similar fashion.

But I'm not the person driving that.

goldfirere commented 1 year ago

I think this is all fantastic! Thanks, @hasufell, for writing up your detailed start of this conversation and for being a motive force here.

One challenge I have thinking about this concretely is that it's hard to pull out the proposals from the problem statement. That is, I see a proposal for a synchronous coordination meeting in advance of releases. But, other than philosophical urgings (which are important!) I don't see much else. Maybe I missed something (I likely did), but that's why I'm writing this paragraph, as a request to pull out the proposals from the rest of the text.

I do see the idea of creating a shared project. Might I suggest placing this at https://github.com/haskell/haskell? That's some prime real estate that has yet to be used. Its front page (that is, README) could contain a little map of our ecosystem, along with instructions for contributing (including where to post bugs, what communications media to announce new libraries in, our standards of discourse, etc).

I urge us not to get distracted by Nix. My assessment is that Haskell has currently roughly saturated the market for early enthusiasts, and that this proposal is part of the broad push within the Haskell community to "cross the chasm". (The first 20 seconds or so of this video explains the term well; excuse the marketing for a specific book, which I have never read and am not advocating for.) The existence of the Haskell Foundation can be seen as another part of this broad push. I don't have a handle on whether Nix is also trying to cross the chasm, but my assessment there is that it hasn't done this step yet: it is a powerful tool for enthusiasts (and people who pay enthusiasts to help them), but I do not think it is yet ready for mainstream. By tying our efforts here to Nix, we will be trying to connect two enthusiast-oriented technologies and making both easier to use at the same time, which seems much harder than doing them one at a time.

hasufell commented 1 year ago

That is, I see a proposal for a synchronous coordination meeting in advance of releases. But, other than philosophical urgings (which are important!) I don't see much else.

There are 6 proposals:

  1. binary distribution metadata: https://gitlab.haskell.org/ghc/ghc/-/issues/22686 (to be implemented in GHC)
  2. synchronous release meetings (joint effort)
  3. Making it feasible to change bindist mappings
    • this needs to be implemented in cabal
    • some GHC issues are related too
    • some other parts are GHCup issues
  4. Developing a holistic tooling release workflow
    • some parts of this are already WIP by me
    • GHCup will likely augment the ghcup-gen program to facilitate generating metadata updates based on certain input
  5. Creating a shared project for tooling
    • a joint effort
  6. Increasing the bus factor across critical tooling
    • something the HF needs to think about

I think all of these proposals have clear action points and scope, while 4. is maybe a bit more vague. What exactly do you find missing?

The specific immediate goals of all those proposals are to:

hasufell commented 1 year ago

@goldfirere to be clear... this isn't a formal proposal that expects a vote by the board.

I'm using the HF issue tracker to raise this for visibility and also communicate the points where HF can get involved.

The decisions on what to do in particular depend on motivation by volunteers and communication and discussion between the stakeholders. The technical parts depend ultimately on the respective maintainers who have authority over their projects.

goldfirere commented 1 year ago

What exactly do you find missing?

The summary you have now provided. That's really it! The initial post was long and detailed (and thus forms a great base of shared understanding and conversation-starting), and I failed to extract the concrete changes you were proposing.

Thanks for writing that summary -- that's really all I was looking for. I remain strongly in favor of this direction of travel.

hasufell commented 1 year ago

That's really it! The initial post was long and detailed (and thus forms a great base of shared understanding and conversation-starting), and I failed to extract the concrete changes you were proposing.

The idea is that if I'm hit by a bus tomorrow, someone else can pick this up and go back to this post to understand what's going on.

I realized while writing it that it's indeed not that easy to explain all the context properly. Which is a strong sign that it's important to do, so it doesn't just live in the inner circle of maintainers who happen to deal with the bug reports.

chreekat commented 1 year ago

Thanks for writing this! You've hit the nail on the head by focusing on end-user experience in tooling. If we don't agree on the problem, we won't agree on the solution. I think we agree on the problem.

I just want to say that I hope my role (Haskell Foundation DevOps) is considered as a resource in this area of effort. In fact, although my main focus is still GHC CI, my current task is literally to improve the Mac bindist, so I'm already contributing in this area.

I am resisting putting on my project manager hat (gotta code fast!), so I will mainly just lurk for now...

chreekat commented 1 year ago

binary distribution metadata: https://gitlab.haskell.org/ghc/ghc/-/issues/22686 (to be implemented in GHC)

Ben has already scheduled this for the 9.6.1 release, i.e. the next one which is coming out in a couple months.

Is it needed for the other, older supported releases? Does it need backports to be useful? Or is it sufficient that it will be available for 9.6 and onwards?

david-christiansen commented 1 year ago

I am happy to support this initiative. A holistic approach to users' experience is important, and the historically-determined boundaries between various Haskell projects are not necessarily where lines would be drawn if we were to design everything today.

Ericson2314 commented 1 year ago

The historically-determined boundaries between various Haskell projects are not necessarily where lines would be drawn if we were to design everything today.

Love it. I think rectifying this the unifying theme of all the key meaty projects that the Haskell Foundation is here to help us fix.

hasufell commented 1 year ago

@mpickering and I talked about:

  1. improving communication between tooling stakeholders and how to deal with release workflow
  2. improving CI and automation

I had the feeling we're on the same page. I think the next steps here are to gather more concrete ideas of how we structure these things (when to have meetings, how to adjust release workflow if necessary). Matthew took ownership of driving this forward initially.

We also talked about:

  1. shared space for high-level issues that spans multiple tools and affects end users
  2. where to put documentation for tooling issues that can be consumed by end users (related to https://gitlab.haskell.org/ghc/ghc/-/issues/21491)

So right now, I think we're trying to figure out the right structure.

My impression is that my proposed survival guide, the HF error project and a shared tooling project that tracks high-level issues (such as the GHC ABI one) are all connected in some way.

For the tooling project my idea is that:

I'll pursue further input from HF and stakeholders about 3. and 4.

Ericson2314 commented 1 year ago

all issues are motivated by improving the end-user experience rather than some technicalities

I get it, but we also have a lot of arcane technical debt that then indirectly bubbles up as user experience issues. Just knowing the pain points does not uncover this.

hasufell commented 1 year ago

all issues are motivated by improving the end-user experience rather than some technicalities

I get it, but we also have a lot of arcane technical debt that then indirectly bubbles up as user experience issues. Just knowing the pain points does not uncover this.

Surely.

The scope of that issue tracker/project space is about reflecting on the tooling experience.

Not so much how to improve Haskell or GHC or base etc.

One might arrive at ideas from either direction: seeing user bug reports and thinking about the causes or seeing technical issues that (as you point out), will bubble up to the user.

I'm just trying to make clear that this project space really has a very specific scope and isn't meant to compete with HF tech proposals, base proposals, GHC proposals etc.

I think it's hard to work on Haskell tooling as a holistic experience if all the projects are disconnected.

david-christiansen commented 1 year ago

I just had a call with @hasufell. We hit on a way forward with this that seems workable. The plan is to create a space in which cross-cutting tooling issues are documented in a way that takes concrete user experiences as a starting point.

For example, the documentation for an issue might begin with 'A user opens a Haskell file in VS Code on their Mac. The following error message occurs when they click "Build"'. This would have screenshots and enough details the reproduce the issue. After that, there would be a summary of the technical issues that cause this error to occur (e.g. that the error occurs because of ABI compatibility issues between tooling, or that it's due to code signing requirements from the OS, or that the user didn't have Internet access at a given moment leading to partially-installed tools - these are just for the sake of example). Following the summary should be any known work-arounds for the user. After the workarounds, the issue can then describe the specific details of what's causing the issues and link to relevant issues in the issue trackers for all the involved projects.

In other words, the structure is:

  1. Story - "This was expected, but this happened"
  2. Summary of cause - "It's because of this technical thing"
  3. Workarounds - "How can the user make progress while we fix it?"
  4. Details - "How do we actually go about coordinating a fix?"

To avoid announcing a big initiative that results in nothing but statements of support followed by a subtle "whoof" sound, @hasufell and I will build a bit of example content centered around a real issue or two that have been repeatedly encountered. Having such a text to start with can serve as an example to other contributors as well as guide choices about the concrete technical platform on which to host it. @hasufell will write the initial text, and I'll be a second set of eyes.

Thanks for a great initiative and call, @hasufell!

michaelpj commented 1 year ago

I wonder if @tomjaguarpaw has thoughts, I remember he had https://github.com/tomjaguarpaw/tilapia which I think had similar goals to what you're describing.

I'm a little worried about adding another way of reporting issues or whatever, because it inevitably gets overlaid on top of everything else. If we do something like this, it would be nice to think about how it can reduce the use of existing systems. For example, I could imagine us saying something like "Issues which are opened on the HLS issue tracker but which relate primarily to tooling (?) should be moved to this other issue tracking location" (straw example, quite probably not what we want to do). That would actively reduce load on the HLS issue tracker, which would be nice.

My other $0.02 is about Nix, as a user and general Nix partisan. I think we don't need to consider much in the way of Nix-specific issues issues, because most issues that affect Nix on the tooling side tend to be generic and things that most people would consider to be improvements even if they're not being directly affected by them (e.g. lib:ghc being reinstallable). But we might want to prioritise some issues that would otherwise not seem that important if they would make life a lot easier for Nix users.

hasufell commented 1 year ago

I'm a little worried about adding another way of reporting issues or whatever

This is not meant for user bug reports. This is meant as a documentation and collaboration effort of the maintainers and collaborators of the core Haskell tooling. The intention is to align on shared goals, increase visibility of user-facing issues beyond project boundaries and have a place where resources on complex topics can be gathered in a refined way that connects all the small pieces together.

Depending on engagement, it may also be the place where we dump meeting notes of the synchronous meetings, develop elaborate guides on how to deal with tooling issues (aka survival guides), develop ideas about user experience etc. It will not be a substitute of individual issue trackers.

It may also serve as a foundation for project pitches for actual tech proposals or requests for funding.

For user feedback, we may utilize the "Discussion" feature of GitHub.

These are all ideas and all of this depends on the eventual engagement we generate.

Ericson2314 commented 1 year ago

I wanted to make a small announcement that after a long while of being stuck, I was able to make more progress on https://gitlab.haskell.org/ghc/ghc/-/issues/17191 and hope that I am finally "over the hump" with an incremental path forward for the rest.

I bring this up as an example of something that is utterly divorced what I imagine are most user would complain about, but as something that nonetheless informs a lot of that as a root cause, and certainly benefits from coordinating between tools/teams. For example, this is one things that informs where the bindist metadata ought to reside. It gets more interesting with follow-up like @bgamari's WIP https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9263.

angerman commented 1 year ago

Thank you @Ericson2314 thid has been overdue for a very long time 🙈

hasufell commented 5 months ago

Closing. This wasn't a proper proposal to begin with and a half baked way of saying "we need haskell tooling product management".

After thinking about it for a long time, I don't think we're in a position to achieve that currently. And I don't know how to propell that idea.