Improvements to error messages and their infrastructure

[goldfirere]

Update: this has been superseded by https://github.com/haskellfoundation/tech-proposals/pull/24. Original proposal follows.

---

This proposal sets out an agenda for improving Haskell's error messages. It seeks coordination and consistent energy from an HF person as a way of powering volunteers.

Rendered

(Note: I propose this solely as an individual Haskeller, not in relation to my work with the HF. As that individual Haskeller, I think this is a nicely defined project that would have some real wins. As an HF board member, this looks like a plausibly good use of our time and resources, but there are many other plausibly good uses of our time and resources. I don't claim any particular insight that this is the best use of our time and resources, and understand this may reasonably be rejected. I defer to the wisdom of the HFTT members on this point, just as any other proposer would.)

[gbaz]

I'm glad there's progress in thought on this front. I would like to see a proposal for a general convention for "identifying the speaker" which we could coordinate adhering to across tooling. A simple, concrete spec in that regard would be very welcome and I think work as a standalone proposal -- I suppose in this repo, for lack of a better location.

The "what we need" section seems a bit all over the map and then performers just specifying "to help push this work forward" is somewhat vague. I'd support starting a bit more concrete and narrowing down to just point 4 of "what we need": "The error-datatypes work has already created lots and lots of error-message constructors. Not all of these are well documented or exemplified." The proposal suggests that this can be somewhat "crowdsourced". Is the proposal that this is all through individual PRs to GHC? Perhaps there should be a branch that only focuses on the "types" file and then an army of volunteers can be coordinated to just build that branch up into a single PR? As an individual project it seems manageable in scope with decent returns. A lot else still feels too vague to be actionable.

So I think you'll find that lots of people really like this direction. But in terms of what makes a good proposal, that in my mind requires signing off on actionable things with a plan to make them happen, and there's just too much here, in too many states of flight, not all of which is clear what foundation resources would be directed to what ends, and I think that needs a lot more sorting out before it makes sense to say "yes" -- I want us to approve proposals that can be moved forward ones, not ones that are so broad that they somewhat fall apart. In other words, I very much appreciate the big picture, but I think we need some more specific plans of actions and direction for allocation of resources if we want to make this a good proposal.

[goldfirere]

It's true that this proposal is open ended. Does that automatically make for a bad/vague proposal? I hope not. It's meant to suggest a direction of focus for the HF, with concrete tracks to explore.

Re-reading the "What we need" section, all of the points look actionable to me. Some (1 & 2) are perhaps more reactionable, meaning that the point is more that the HF source someone (perhaps a dedicated volunteer) to be an active steward of an existing resource. Which of the points are unclear to you? I can try to clarify in the text.

It's all really seeking technical (and social) coordination, much more than, say, coding. If that's out of scope here, then do you know of a better place to put this?

[emilypi]

Did this have a Pre-HFTP on discourse we can cite?

[davean]

I like the "what" but I'm lost as to the "how" being outlined. If we approved this what would that mean other than we think this is neat?

Also, I hate to be that person but this doesn't seem to have followed the proposal process - particularly that under "Initial Discussion". I'd think it would be better if we just made it clear there's exactly one place but that's not the procedure we have.

[Ericson2314]

If I may plug a bit more broadly the topic of "actionability" and scope in general, something I have been discussing with people is that the current proposal process is somewhat limiting in that proposals must be big enough to warrant marshalling dedicated resources, yet small enough that they don't seem too nebulous or risky.

Those conditions yield a very small goldilocks range, one already misses out on engaging with the smaller-scale forms of resourcing the HF already does, such as organizing in-kind donations of developer time.

I am hoping we can off a more split system where good ideas can get approved planned out in an official backlog, and then resourcing at all levels can happen based on that back log. That separates the "what we want to do" from the "how we want to do it", and in particular means we can alternate the size of the effort / scale of the risk over time as the current actionable "fringe" of the backlog changes.

---

I bring this up because I think that "what" vs "how" separation would be very advantageous for this just of goal. While there are plenty of open questions, the first step are (to me at least), seems pretty clear:

Scale up, throwing more bodies at converting to today's sort of error message data types

This is a nice task because it is parallel, with each PR being as small as we like, and relatively little onboarding required as GHC tasks go. It thus is a great example of something that works with flexible staffing.

Once that is done, however, the task gets more nebulous, and it would be good to scale back down / transfer people to other tasks as we plan out how to do the next tasks.

As I wrote in the comment of https://github.com/ghc-proposals/ghc-proposals/pull/307, for example, I fail to see much value of it once https://github.com/ghc-proposals/ghc-proposals/pull/306 is done. 307 is overkill for the GHC executable in its current form, and underkill for the IDE which will probably want to work from the 306 fully structured errors directly.

Going further, on the topic of the "you probably don't want to enable -XDataKinds" example in this proposal, one thing that's bugged me for a while is that the code disabling extensions in GHC is very ad-hoc and unprincipled. https://github.com/ghc-proposals/ghc-proposals/pull/476 is a good example of this, where @goldfirere points out the code to disable -XPolyKinds is just icky and a maintenance burden and that's why he was hoping to just delete it. I'm kind of a "worse is not better" purist here, and think the very notion of GHC supporting a combinatorial languages needs some research so these languages in ideal form are more principled / self-contained, and this is a hard prerequisite to having truly great error messages.

So in this case, not only should the HF scale back, but it might kick the problem back from "development" to the research" side of "R & D". Only after those questions are figured out would I want to scale back up to the same level of effort converting structured error messages warrants.

---

I think this pattern of large projects that are best not resourced as one long effort, and being a mix of a low hanging development and high hanging research, is actually the common case, and perfectly health. Furthermore, it's a framing that we as the "research and industry together" language should be exceptionally well-positioned to execute.

I therefore see changing our process to do that "what" vs "how" separation as less a burden in response to the issues we face being hard, and more as an opportunity to play to our unique strengths as a community.

[goldfirere]

Did this have a Pre-HFTP on discourse we can cite?

Yes: https://discourse.haskell.org/t/pre-hftp-improving-infrastructure-around-error-messages/3545

It got a few likes and no responses. Maybe that means this is direction is not of interest to the community. That would surprise me, but (if we can somehow confirm this) may be a reason not to pursue this further. I think that, in this case, it's more likely that I didn't, say, tweet about the pre-HFTP or otherwise draw any attention to it.

If we approved this what would that mean other than we think this is neat?

We would start engaging on all the points in "What we need", maybe all at once, maybe not. I've clarified the text in this section, delineating what I propose the HF take on for each bullet point.

@Ericson2314

I'm afraid I don't see concretely the interaction between disabling extensions and structuring error message text. I do see how the phrasing of some error messages might depend on the design work you're talking about, but I don't see that it would impact deeply the design of a structured error message type.

[Ericson2314]

@goldfirere err I think I am staying as concrete actions to take they don't interact. We can very simply today just convert everything to structured errors, as we should do that.

My point is that after we do that, it's not clear to me or there is are good next step the HF is well-positioned to help with. But that's fine. The HF can take a step back, see how the uptake of the structured errors is going, and plan accordingly.

[gbaz]

Does that automatically make for a bad/vague proposal? I hope not. It's meant to suggest a direction of focus for the HF, with concrete tracks to explore.

Right. I think there are a lot of potentially good ideas here, but I don't think that the tech track is a place for "a direction of focus for the HF" -- that's a board/executive level decision. Proposals should be about actionable allocations of HF resources. Lets leave aside exactly what resources are available here, in terms of people, and think about actionable items. I think as a proposal (not as a direction) this would benefit from being cut down significantly to just one or two actionable things, with clear plans. So a proposal on specifically crowdsourcing finishing up the structured errors in GHC stuff would be great -- and an allocated person could do good work in generating a list of places that should be examined to find errors to convert, a list of errors that need documentation, a guide for others to do the work, etc. In fact, as I was going to sleep last night, it hit me that this could be the right sort of thing for an HF organized "virtual hackathon" -- someone with experience prepares tutorial material on how to help, and places for contribution -- a spreadsheet or something documenting what needs work, (and which others can add to), and a tag for collecting PRs. Then people, based on the presentation, organize to either find things to add to the spreadsheet, or tick off tasks from it, with the people running the hackathon consulting and advising. So now we have something very concrete -- organizing an event with associated materials to help leverage a group of volunteers to do this error work, and in the process, educate a lot of people on how to better contribute to GHC errors.

We could couple to that a proposal that an appropriate person then organizes cross-tool discussions and writes a spec for structured error reporting (perhaps flag-driven?) across tools. But for that, we really need to have the right person in mind, otherwise its just nice sentiment that's a dead letter. Ideally this would be donated time from a high-level engineer from some Haskell-involved company. But that should be in the "performers" section.

So, again, I want to emphasize that I'd like to see us find a way to take the good ideas here into an actionable proposal, but that's going to take stepping back and thinking about how to extract what is concrete, and maybe develop that further. In my experience, a plan that says "we should coordinate" is almost always a dead letter unless its coupled with a specific direction in how to coordinate, which specific things should occur, etc. The hackathon I suggest here is not the only or necessarily best idea in this regard, but it provides some sense of the direction I think we should be looking towards.

[goldfirere]

@Ericson2314 Ah -- then we're on the same page. I agree.

@gbaz

I understand better now. It sounds like you're saying proposals shouldn't just have the "what" (e.g. "source volunteers to complete goal X on project Y") but also the "how" (e.g. "organize a hackathon" or "give out free stickers" or something).

Yet, at the same time, what-vs-how is a sliding scale. The proposal could say "I want better error messages" (and that's it). That's a what without a how. But the proposal goes significantly further, requesting stewardship of the error-messages repo, volunteer coordination for documentation and error-message conversion, and oversight of a project to refactor GHC to have structured error messages. That all is a how, to me. Of course, you could classify all that as a what, with the how being stuff like hackathons or a volunteer-o-meter or a particular flow chart someone could follow to try to bring discussions to a useful conclusion. Then again, those hows could also be reclassified as whats, as they miss details such as where to advertise, what venue to hold a hackathon in, etc. My point here is that I find the what-vs-how classification too much of a movable feast to be used as a way of evaluating a proposal -- or at least too much of a movable feast for me to feel confident that any change I make addresses your concerns.

Instead, are there particular details that are missing here? I think being specific about questions will help move this forward.

Note that there is a suggested resource allocation of 5 hours per week. I could write out how those hours could be used (at least at first): going through the error-messages repo, looking for unfinished conversations, and posting there constructively to get them moving to a conclusion; building, say, a spreadsheet in which to track progress toward error message conversion; looking at other language communities to find exemplars for an error-message directory, and starting to design such a thing; going through the history of structured error messages in Haskell and in Idris to try to work out a design, reaching out to potential clients such as HLS. To me, these steps are implied within the text. If they are unclear to you, then I have an opportunity to make the text better.

[gbaz]

Right. None of those steps are implied within the text to me. Also, I really would cut out the error messages repo as part of this proposal. I think the "high leverage" part of this proposal is in building out the infrastructure to enable changes, etc not to shepherd individual error discussions.

The "what vs how" part is to me -- suppose I was given this proposal and told "ok, go do this" -- what would I do? Also, a lot of what you're proposing is "somebody should design a thing". I think that designing things without much to start with is not a good proposal -- a proposal should start with at least a skeletal design in place, and be about someone implementing that design. Alternately, a proposal for a design should say specifically, "we want a design that hits these specifications and solves these problems" and we want to source someone to design it -- so even if you don't have the specifics of a design, you at least have sort of given guidance for the key elements of it. I.e. if you think there's work in other communities that can guide what an error message directory looks like, the proposal should point to that work, discuss it, talk about how it can be adopted, etc. I think that we should have something in a state where someone can read it, and think "ok, I know how I can go about doing this" and hit the ground running. I agree "what vs how" is a sliding scale -- I don't have a clear and simple rule on how far into the "how" we need to go -- I just know its not enough yet, and I'll know it when I see it :-)

[Ericson2314]

I want those things @gbaz mentions, but I don't think I want them in atomized proposals. The first part, structured errors, I view as highly synergistic with other efforts to improve the data structures in GHC more broadly, such as the stuff @hsyl20 and @doyougnu have been working on. All of this is fairly amending to outside contribution, though the breaking up of DynFlags and HscEnv is quite a bit harder, it is still more blood sweat and tears and needing some big fundamental insight.

If we put all the details plains in separate documents structured by the high level goal, we were going to loose out on the interplay, and miss how a family of yak-shaves very remotely related to the end mission of the HF actually does in fact get us closer to multiple goals at once, unlock more "itch scratching"-type contributions, etc. etc.

Similarly, even saying 5 hours a week is premature at this point: first we want the work broken down into actionable steps with concrete dependencies, including the interplay with other projects in execution or planning stages, then then we decide what level of resourcing is appropriate. Putting both those steps into one proposal just means we are going to be arguing at multiple levels of abstraction at once in a big unwieldy GitHub thread.

[Ericson2314]

To be really concrete, @goldfirere has a bunch of possible steps here, and I think the first task ought to be debating whether those are good. For example, I would say

1. 306 structured errors is good, and good for the HF
2. 307 richer SDoc is not so good
3. Social and ecosytems steps seem good, but unclear what the role of the HF should be.

If others agree, 1 and also 3 would go in the backlog document, because it is good to track even work that the HF might not do, if other work that the HF does do relates to it. On the other hand 2 wouldn't if we are skeptical it is a good idea regardless of who does the work.

After that, we can go back to resourcing and say "hmm, only 1 is actionable, let's make a resourcing proposal for just that." We decide, e.g. what combination of putting HF managed devs directly on the task vs shepherding volunteers is appropriate --- there is still plenty to took about just for resourcing alone! The important thing is we have this conversation having already decided that 1 is a good thing to happen, and we can refer to the backlog version of this task, and that prior discussion by which it earned its inclusion in the backlog, as shoe-in motivation.

Structured errors alone is just a random technical step. Without the context of "this helps IDEs, etc. etc." in the ghc-proposal, it isn't motivated. It doesn't even standalone, for someone needs to do the IDE work to make it pay off. At the same time, it is really good as even the sole step the HF can take on this front in the short term, because the other steps have issues, and because we are confident the IDE team is on board and ready to self-manage translating the structured errors into the end "use value" of better IDE errors for regular Haskellers.

I don't think anything but a split planning and resourcing process and properly do justice to the fact that on one hand HF should probably execute narrowly at this time, but on the other hand, the structured error messages just one part of a broader and quite nuanced plan of trying to improve the IDE, and have it supplant "raw GHC/GHCi" as the primary UX for Haskellers that the HF is focusing on.

[gbaz]

@Ericson2314 You're right that the proposals scope is limited in what it can accept, in that it should only be accepting things that can actually be executed, and not things that one wishes could be executed. I would ask that, for now, for this proposal, you accept that limitation rather than trying to simultaneously change it and also discuss proposals on the table in front of us in the same thread.

I think that we are new enough, and have limited enough experience doing things and getting them done that this limitation is fine, and in some ways good, because it forces us to focus. As things develop and scale up, we can consider trying to grow this.

Note that a "backlog" can also be proposals that people agree should be accepted, but can't be resourced at this time, etc. And note that the "overall picture" for something big like "all haskell ecosystem error messages" can be kept track of in some place outside the proposals process. Since the product of the proposals process is a static document accepted and merged as a completed proposal, that sort of ongoing tracking is arguably not suitable for here anyway. Our goal is not to try to do everything, all at once. For now, let's try to get this one thing right, a few more times, and then step back and reassess. This is to say that yes, planning should happen on a bigger scale than the proposals process -- but also, I do not think that big scale planning is something the HFTT as such should try to manage at this time. We haven't even proven ourselves good at just doing the narrower proposals process that much yet. So let's stick to that for a while :-)

So I would like to keep discussion here focused on how to guide @goldfirere to produce a top-notch proposal that the HF can execute on and yield good results. And, for now, I think that means also trying to focus this proposal on a few points that are most actionable and have the best effort/impact ratio, and hang together in a single project.

All that said, I agree with you that 1) is good and also actionable, and I think that is a good focus. I also lean towards some version of 2) being potentially good, but also I think it needs a lot more work to sketch out what it would look like so we can even agree if we're thinking about remotely the same thing -- otherwise, who knows.

[Ericson2314]

I'll shut up about process now, but if the point is doing the simplest things while the HF is getting up and running, and also to have stuff to do so we can get up and running, then I will more firmly opine this proposal should be cut down to just fund ghc-proposal 306. 306 being accepted and futher (edit actually closed as out of scope, but was more popular before closing) validated by the work in progress makes it safe.

If we say we don't yet have the capacity to plan as an organization, we shouldn't try to wing it and fund a unplanned projects anyways, but simply skip unplanned projects while there are planned ones. Continuing 307 in the ghc-proposal thread can work as a planning mechanism that doesn't burden HF, and if that ghc proposal is accepted we can happily consider another small-simple HF proposal to fund it in addition.

Does that sound fair?

[gbaz]

My understanding is those ghc proposals are somewhat superceded and the current status of things is documented at https://well-typed.com/blog/2021/08/the-new-ghc-diagnostic-infrastructure/

Which in a sense makes this all more confusing -- the proposal before us describes a lot of work-in-flight in various capacities. I'm still not sure exactly what concrete things having a slice of dedicated time from HF would be best served on pushing on, and how they might be sort of wrangled into a self-contained project. But Richard (and I suppose Alfredo Di Napoli, and a few others) are the experts here. And Richard says that he already has some ideas about how to make things concrete. So I don't want to speculate about moving drastically away from the direction he wants. I want to give him our feedback about what feels too vague or not as high impact, and what seems most promising, and then let him rework the proposal (and I'd be happy to hop on a call and chat, or others could) to try to explain that stuff that he thought was implicit but others didn't understand, and streamline things, and see what emerges.

[Ericson2314]

Sure if they want to provide more plans and details, that's fine. But I don't want them to feel like they are on the hook to plan further out than makes sense right now just to get started on anything at all.

https://well-typed.com/blog/2021/08/the-new-ghc-diagnostic-infrastructure/ doesn't really mention 307 --- great! It has:

1. Completing the refactoring
2. Haskell Language Server integration
3. GHC unique diagnostic codes
4. Diagnostic message plugins
5. JSON output from GHC

2 I think is the most meaty of those and the most useful --- after all if the IDE doesn't end up using the structured errors this was all in vane.

So in the event that finishing 306 alone is too small for some reason, that is a good option. Per https://github.com/haskell/haskell-language-server/issues/2014 it seems a large part of the work is simply getting/keeping HLS building with the latest GHC HEAD API. In fact, that alone could even be an on going HF task, akin to keeping GHC running smoothly, for we better keep breaking the GHC API until it is not atrocious, yet that will unfortunately create busy work for HLS in the meantime, even as HLS is one of the greatest beneficiaries in the long run. We therefore have between just 1 and 1+2 many options for how much/long we want to resource this.

Conversely, with 4 and 5 I think the benefit is marginally, and if someone really wants this it shouldn't be hard to scratch the itch on the shoulders of 1. 3 as we discussed is good, and also not to much work, but seems like it needs more an ecosystem-proposal to rally the community with a little bike-shedding fun :smile: than HF pushing the implementation at this juncture.

Again, I am not trying to tell @goldfirere, @adinapoli, et al. what to do. But insofar than it appears that there was some confusion on what the current process entails, I just wanted to throw out concrete examples of existing plans that seem to me to be a good fit. I thought examples might be easier than finding general rules, too, given the what-vs-how discussion above. Of course these are just my opinions and they are are free to disagree if they rather take the concrete planning in a different direction.

[goldfirere]

@Ericson2314

Again, I am not trying to tell @goldfirere, @adinapoli, et al. what to do

Please do tell me what to do! That is, the proposal text has a number of ideas. I think they're good ones. The various feedback I've gotten on this thread and elsewhere suggest that some of the ideas seem to have more value than others. This is exactly what I was hoping to get out of this process -- broadening the brainstorming from one person to many, and weighing which parts of this are likely to have the biggest impact. I already see specific threads coming out from all this, and I will use these to revise the proposal (next week).

Maybe, in retrospect, this is why I didn't jump on the "make this more concrete" push at the beginning of this thread. The concretization (the how) is important (and I now see how I can do this), but weighing which goals are most worth pursuing needs to come first. This is already happening, but I just want to explicitly welcome this kind of feedback. I can then restructure the proposal to put these best ideas first, maybe relegating the other ideas to an appendix or not at all. (My thought is to keep them somewhere, in case we circle back later.)

[Ericson2314]

@goldfirere Well, glad that it is in fact useful to you! I guess that cue for anyone else with different opinions to start chiming away so mine don't have outsize impact!

[gbaz]

We had some general discussion on this in the HFTT meeting, bearing in mind that we are waiting for a revision to have a further discussion. I encouraged people to comment individually, but in my estimation the theme seemed to be that structured errors for different tooling were the biggest win, and figuring out how the HF-as-such could contribute to this was a central concern. There was also definitely keen-ness on associating error messages with potential hyperlinks to pages that could teach users how to interpret such messages. Since my summary is based on what I most remembered, which is likely what I most agreed with, I would again encourage others to drop a little feedback here as well :-P

[goldfirere]

Just in case anyone drives by here: This proposal needs revision by me, but I have not had the time lately to put into this. If it's helpful to the HFTT, feel free to close; I will reopen when I'm ready to put some dedicated time into this. Otherwise, I hope to return here soon.

[Ericson2314]

Speaking personally, if you simply mark it as draft, that seems good enough to make it both accessible to others while preventing it from gumming up any queue.

[Ericson2314]

@goldfirere would you have any time (perhaps with assistance?) to push on this over the next week? We have an HFTT meeting coming up on 2022-02-02, but an empty agenda so far I fear.

I think this proposal, shrunk down as discussed, would be perhaps the most actionable proposal based on the sentiments last meeting, but if we miss that meeting then it will be stuck in limbo for another month.

[goldfirere]

Yes, this is good timing, and a good prod. I will aim to have a revision by Wednesday night.

[goldfirere]

This proposal has been superseded by #24. I could have used this same PR, but someone reading the new proposal and then the commentary would be (understandably!) confused. Given that #24 is a complete top-to-bottom rewrite, I decided to move to a new ticket.

Thanks, all, for the suggestions thus far.