Track LanguageServer exceptions

[fm-117]

To improve monitoring of our LanguageServer, we have to introduce some telemetry events for server exceptions. This issues encompasses multiple goals :

• introduce support for telemetry/event LSP method
• streamline exception handling in LanguageServer
• categorize exceptions that require a window/showMessage opposite to telemetry/event
• define exception serialization and implement telemetry event sending

[fm-117]

LoggingSystem is also part of the 'problem', we should improve the current implementation by using an AutoResetEvent instead of a periodic poll.

[fm-117]

Current LS observations:

• LanguageServer.NotifyWarning and LanguageServer.NotifyException are the methods dedicated to signal a "Warning" or an exception to the client
  • NotifyWarning is almost irrelevant as it is used only in TypeCobol context to warn about problems during intrinsics or dependencies loading. This is not used in pure-Cobol context.
  • NotifyException is the main target of this issue, the goal is to ensure a telemetry/event is sent to the client when it is called
• NotifyException is used in many contexts:
  • in each notification or request method: each notification or request method (beginning with Call or Received) uses the same pattern and have a try-catch with a call to NotifyException in the catch block.
  • The JsonRPCServer methods: HandleNotification, HandleRequest and HandleResponse also have a catch block calling indirectly NotifyException. This is redundant with the pattern mentioned above.
  • NotifyException is called on the AppDomain.UnhandledException event
  • NotifyException is called when the semantic update timer thread crashes
  • finally NotifyException is called by TypeCobolServerHost.MessageHandler to protect agains exception happening when executing an auto-message (Action instance created by the server to perform work on its own message queue).

We should cover all threads working simultaneously in the server to ensure the application won't crash and also track all exceptions. In our LanguageServer, we have at most 5 concurrent threads:

1. main thread is responsible for reading the command-line options, starting the worker thread, and listening to incoming messages and putting them on the queue
2. worker thread is responsible for reading the pending messages on the queue and process them one at a time, in order
3. background compilation thread is responsible for refreshing the AST of the current document 750ms after the last didChange event received
4. logger thread is not part of LS but is activated by LoggingSystem, it is responsible for performing logging actions
5. message sender thread is responsible for sending messages to client. While it could be done by worker thread, the current implementation delegates the actual sending on another thread

Following table states how each thread is protected against crash:

Thread	Status	TODO
main	not protected directly, but a crash will be caught by UnhandledException event	Check if some exceptions currently caught can be notified
worker	All client messages are covered by 2 catch blocks, each having a notify mechanism, auto-messages are covered too	Simplify code ? Avoid redundant catch blocks ?
background compilation	Covered by a can block + Notify
logger	Not covered but it's ok, logging is secondary and should be considered as safe	Improve implementation using AutoResetEvent
message sender	Not covered	The ThreadBody does not have its own try-catch block, waiting the sending task is alsso unsafe

Unlike other locations, AppDomain.UnhandledException denotes a fatal problem: a window/showMessage should be sent to the client to signal the server has crashed.

[fm-117]

Proposal for event serialization:

        public class TelemetryEvent
        {
            public string type;
            public Dictionary<string, string> data;
        }

• a string field to define the type of the event and help the observer (client) to deserialize the associated data
• a name-value collection to transfer meaningful data about the event

For example, for an exception, data would contain:

• "Type": "System.ArgumentException"
• "Message": "Invalid argument for param 'example'"
• "TargetSite": "TypeCobol.Compiler.SomeClass.SomeMethod(ExampleType)"