Support structured diagnostics

[dylan-thinnes]

Addresses https://github.com/haskell/haskell-language-server/issues/2014

Should hopefully enable https://github.com/haskell/haskell-language-server/issues/3246

TODOs:

• [x] Resolve TODOs in code
• [x] Add diagnostic codes to more tests
• [x] Make PR on GHC side so that we can eventually remove ghcide/src/Development/IDE/GHC/Compat/Driver.hs
  • Issue https://gitlab.haskell.org/ghc/ghc/-/issues/24996 and MR https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12891
• [ ] Deal with CPP to make this work properly on all versions
• [x] Fix hls-plugin-api using _code field set by withWarnings to determine which warning to use.

Additional work (future PRs):

• Replacing regexes with structured diagnostic checks
• Improve typing so that it can be known that an error has a structured diagnostic, instead of defaulting to Maybe

[dylan-thinnes]

Re: deriving JSON instances, the messages often contain fragments of AST and other things which aren't terribly friendly to ToJSON/FromJSON. I tried deriving an instance and immediately came up against

No instance for ‘ToJSON
                         (Language.Haskell.Syntax.Binds.HsBindLR
                            GHC.Hs.Extension.GhcRn GHC.Hs.Extension.GhcRn)’

which makes me think this approach is not going to be easy.

[dylan-thinnes]

Gonna take a break for today, but two big TODOs remain, and a few open questions.

[michaelpj]

Re: deriving JSON instances, the messages often contain fragments of AST and other things which aren't terribly friendly to ToJSON/FromJSON.

This tickled my brain and I remembered that GHC has its own JSON conversion functions! https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/Utils/Json.hs?ref_type=heads

And they definitely have instances for diagnostics, since they're used in the JSON output for GHC, see e.g. here: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11093

Having written this I realise that that only handles one direction, so it doesn't help us much, oh well.

[michaelpj]

They might have another serialization format, though. There's nothing stopping us from serializing them to bytes and the smuggling a bytestring as a JSON string in data :joy:

[wz1000]

I don't think we should serialise the raw data over the wire, instead we should stash it in some sort of Map and send over an identifier/key which we can later use to look up the value from the Map. It requires care to ensure that the values are properly garbage collected when no longer valid though.

[michaelpj]

I don't think we should serialise the raw data over the wire, instead we should stash it in some sort of Map and send over an identifier/key which we can later use to look up the value from the Map. It requires care to ensure that the values are properly garbage collected when no longer valid though.

There's no viable serialization format anyway. I do think it would be nice if there was one, though.

That said, I don't think we should need this info in diagnostics that we e.g. get back from the client. So the main issue is when we e.g. ask for the diagnostics covering a range so we can then match on them to work out code actions. At that point we need the GHC diagnostics. So I don't think we need a separate store, at worst we might have to augment our server-side diagnostic store.

[dylan-thinnes]

One remaining point to address from review: https://github.com/haskell/haskell-language-server/pull/4311#discussion_r1651757991, have made a commit for it https://github.com/haskell/haskell-language-server/commit/414c845a74ef64ded31c7486607aa90004bab56e

I'm looking into the CI failures now - I assume some CPP shenanigans are at least part of the issue.