[Update]: Commit new error message API and needed refactoring

[KTSnowy]

This PR contains the new error message API for Otterkit, as well as some refactoring needed to support the new API. This provides a modern look for Otterkit's error messages, which should massively improve the developer experience around debugging and fixing code issues, and also provide useful, easy to read information for the users.

Example of how the messages currently looks like:

The current API is based around method chaining on a ref struct, meaning it shouldn't allocate any new objects on the heap (aside from the strings) and should perform reasonably well:

The API allows for the messages to be later extended with additional optional information. The above .WithNote method for example, is optional. Without the method, the error message would look like this:

This should provide a great way to display useful information to experienced developers, while still being easy to read and understand for new developers.

[KTSnowy]

Leaving another example here for future reference. With some extra fine tuning Otterkit can display some great error messages back.

[GitMensch]

As you've asked: that looks nice.

Did you already implemented a switch to return those in JSON and/or XML to improve automated parsing (IDE/CI)?

How would the source reference look like if the code is the result of a COPY REPLACING from a nested copybook?

How would notes with source references look like? Your last example may want to show the original PROGRAM-ID line, for example.

[GitMensch]

For the error display you possibly want an alternative to specify a range of two tokens and possibly also a manual range. The latter would allow to show the missing period on the right place (after the token), otherwise you could create an error token which is one character wide, has a space as text and is inserted directly after the DIVISION token.

[KTSnowy]

Hey @GitMensch.

Did you already implemented a switch to return those in JSON and/or XML to improve automated parsing (IDE/CI)?

Not yet, still need to figure out which JSON format the compiler should return, and also how to deal with the optional parts of the new error messages.

How would the source reference look like if the code is the result of a COPY REPLACING from a nested copybook?

Currently it doesn't work, the compiler can't yet fetch the copybook name for the errors so it displays back the wrong source line and source file.

(Edit: Possible fix for this could be adding another property to the tokens that reference which file it belongs to. That way it won't matter how nested it is in a copybook, the compiler will always know where the token came from)

I'm not sure how to deal with copybooks with the source reference. Should we reference both the line where the COPY is defined and the line inside the copybooks?

How would notes with source references look like?

Haven't implemented notes with source references yet, but I'll work on it today. I'll post an example of it here later today.

[KTSnowy]

For the error display you possibly want an alternative to specify a range of two tokens and possibly also a manual range.

That's a great idea, I'll work on adding those as overloads to the WithSourceLine method. I'll post examples of these here as well.

For the inserted token, I think I have an idea of how to implement that. The inserted token should probably be colored red, right? That way it won't be confused with an existing one.

[KTSnowy]

Hey @GitMensch, here's an example of notes with source references:

Had to do a little refactoring to keep track of the exact token used for the program definition. It seems to work nicely.

[KTSnowy]

The API is still quite simple to build an error with notes to a source line.

We can keep extending it like this to create some amazing diagnostic messages.

[GitMensch]

Possible fix for this could be adding another property to the tokens that reference which file it belongs to. That way it won't matter how nested it is in a copybook, the compiler will always know where the token came from

If it doesn't take too much computing power to track this for each token, then this sound like a good idea - because then you can even drop the Current (), Filename from .WithSourceLine (which would only override the token reference, if still used).

[KTSnowy]

If it doesn't take too much computing power to track this for each token

This should be fine as long as we can make sure that each token will hold a reference to the same string in memory for each file path instead of creating new strings for each token.

The compiler already keeps track of the file names, so the new property could just be a reference to a file in the list of paths we currently have.

[KTSnowy]

Hey @GitMensch, just implemented the token-based file name tracking.

Added a new field to the Token type that holds an integer index:

The index is used in a .FetchFile() method for the Token type:

The method just fetches the file name from the already existing List of file names that the compiler uses for the error messages. This means that there should be almost no extra memory or computing overhead for this (aside from the extra integer and method call):

The index is assigned to each token during the preprocessing and tokenization of each file, this means that each token should know exactly where it came from, no matter how nested the copybooks are.