org.eclipse.lsp4j.jsonrpc.MessageIssueException: Message could not be parsed with special character `“`

[angelozerr]

In LemMinx we provide XML completion based on XSD / DTD. When documentation coming from XSD documentation contains “, the XML language server send this JSON response but when the client (Eclipse IDE / LSP4E) try to parse the message, it throws teh follwing exception:

java.util.concurrent.ExecutionException: org.eclipse.lsp4j.jsonrpc.MessageIssueException: Message could not be parsed.
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at org.eclipse.lsp4e.operations.completion.LSContentAssistProcessor.computeCompletionProposals(LSContentAssistProcessor.java:106)
    at org.eclipse.jface.text.contentassist.AsyncCompletionProposalPopup.lambda$10(AsyncCompletionProposalPopup.java:352)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
    at org.eclipse.jface.text.contentassist.AsyncCompletionProposalPopup.lambda$9(AsyncCompletionProposalPopup.java:351)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: org.eclipse.lsp4j.jsonrpc.MessageIssueException: Message could not be parsed.
    at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleResponseIssues(RemoteEndpoint.java:358)
    at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handle(RemoteEndpoint.java:317)
    at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:198)
    at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
    at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

Here the XSD documentation which causes the problem:

```<p>When specified, this element provides a default context path of the web application. An empty value for this element must cause the web application to be deployed at the root for the container. Otherwise, the default context path must start with a “/“ character but not end with a “/“ character. Servlet containers may provide vendor specific configuration options that allows specifying a value that overrides the value specified here.</p>

[jonahgraham]

Thanks for the bug report. Can you provide the LSP message that fails, ie the json that failed?

[angelozerr]

Thanks for the bug report. Can you provide the LSP message that fails, ie the json that failed?

Yes sure, here the completion item that XML language server returns:

{
        "label": "default-context-path",
        "kind": 10,
        "documentation": {
            "kind": "markdown",
            "value": "When specified, this element provides a default context path of the web application. An empty value for this element must cause the web application to be deployed at the root for the container. Otherwise, the default context path must start with a “/“ character but not end with a “/“ character. Servlet containers may provide vendor specific configuration options that allows specifying a value that overrides the value specified here.\r\n\r\nSource: [web-app_4_0.xsd](file:/C:/Users/azerr/eclipse-workspace2/.metadata/.plugins/org.eclipse.pde.core/Eclipse%20Application/org.eclipse.osgi/538/0/.cp/dtdsAndSchemas/web-app_4_0.xsd)"
        },
        "filterText": "default-context-path",
        "insertTextFormat": 2,
        "textEdit": {
            "range": {
                "start": {
                    "line": 6,
                    "character": 2
                },
                "end": {
                    "line": 6,
                    "character": 2
                }
            },
            "newText": "\u003cdefault-context-path\u003e$1\u003c/default-context-path\u003e$0"
        }
    }

The problem comes from with the value:

...with a “/“ character but not end with a “/“ character...

Please note vscode support that.

[jonahgraham]

This is looking like a charset issue. Not sure why LSP4J is failing on this but VSCode isn't. I don't know how to test your code in place, but I note in your original issue that you are on Windows which has different defaults. There may be a getBytes() in LSP4J that is missing a charset parameter. Do you know if the language server is including charset in the header? Is message you pasted encoded in UTF-8?

[pisv]

Interestingly, a simple test case

        MessageJsonHandler jsonHandler =
            new MessageJsonHandler(ServiceEndpoints.getSupportedMethods(TextDocumentService.class));
        jsonHandler.setMethodProvider(requestId -> "textDocument/completion");
        ByteArrayOutputStream output = new ByteArrayOutputStream();
        StreamMessageConsumer messageConsumer = new StreamMessageConsumer(output, jsonHandler);
        CompletionItem completionItem = new CompletionItem("default-context-path");
        completionItem.setDocumentation(new MarkupContent("markdown",
            "When specified, this element provides a default context path of the web application. An empty value for this element must cause the web application to be deployed at the root for the container. Otherwise, the default context path must start with a “/“ character but not end with a “/“ character. Servlet containers may provide vendor specific configuration options that allows specifying a value that overrides the value specified here.\r\n\r\nSource: [web-app_4_0.xsd](file:/C:/Users/azerr/eclipse-workspace2/.metadata/.plugins/org.eclipse.pde.core/Eclipse%20Application/org.eclipse.osgi/538/0/.cp/dtdsAndSchemas/web-app_4_0.xsd)"));
        ResponseMessage responseMessage = new ResponseMessage();
        responseMessage.setId("1");
        responseMessage.setResult(Either.forLeft(List.of(completionItem)));
        messageConsumer.consume(responseMessage);
        ByteArrayInputStream input = new ByteArrayInputStream(output.toByteArray());
        try (StreamMessageProducer messageProducer = new StreamMessageProducer(input, jsonHandler))
        {
            messageProducer.listen(message ->
            {
                assertEquals(responseMessage, message);
            });
        }

works just fine.

Also, I could find no encoding-related issues by inspecting the code in StreamMessageConsumer and StreamMessageProducer. In particular, there are no occurrences of String.getBytes() (without arguments) in LSP4J.

Are you sure that the MessageIssueException is caused by the special character and not by something else (e.g. https://stackoverflow.com/questions/11484353/gson-throws-malformedjsonexception)? Could you provide the exact steps to reproduce the issue?

[angelozerr]

The problems comes from client side with LSP4E in Eclipse IDE and it works with vscode. XML language server uses LSP4J and Eclipse IDE LSP4E too.

So the problem comes from LSP4J on client side. I'm pretty sure that the problem comes from with the “ , because when I replace with ", it works perfectly with LSP4E.

@mickaelistria could you help us with this problem please.

[cdietrich]

shouldnt this be reproducible with a simple unit test calling MessageJsonHandler.parseMessage

[pisv]

The problems comes from client side with LSP4E in Eclipse IDE and it works with vscode. XML language server uses LSP4J and Eclipse IDE LSP4E too.

So the problem comes from LSP4J on client side. I'm pretty sure that the problem comes from with the “ , because when I replace with ", it works perfectly with LSP4E.

Thanks for the details. That's even more strange, because I cannot reproduce the issue with this completion item using a test language server (based on vscode-languageserver) and a language client for Eclipse (which uses LSP4J 0.12.0, but is based on LXTK, i.e. it does not use LSP4E). The documentation string renders just fine in the UI.

[cdietrich]

does lemmix send a charset in header?

[cdietrich]

also: do we have the cause for the MalformedJsonException it might give a hint where it stops e.g. com.google.gson.stream.MalformedJsonException: Unterminated object at line 6 column 297 path $.documentation.value

[angelozerr]

does lemmix send a charset in header?

We don't write any component which does that,just using LSPJ4 without extra configuration.

I tested in Eclipse IDE && WWD which support TypeScript (it use vscode-languageserver I think), it works but the language server returns:

with a “/“ character but not end with a “/“

[cdietrich]

can you debug lsp4e for the MalformedJsonException to give the cause? you should also see the string gson gets to see there too.

[cdietrich]

i tried to reproduce with lemmix and lsp4e but i failed so far. i dont see the MessageIssueException i wonder if the is OS/Eclipse/System Config/Java Version problem.

[cdietrich]

[pisv]

i wonder if the is OS/Eclipse/System Config/Java Version problem

@angelozerr Can you also try to execute the test case from https://github.com/eclipse/lsp4j/issues/616#issuecomment-1125947942 on your workstation?

[mickaelistria]

@angelozerr Can you please provide files and steps to reproduce this issue in Eclipse IDE with latest snapshots of Wild Web Developer and LSP4E?

[angelozerr]

My bad, you are right it works. In my case I start LemMinx with an Eclipse debug launch (to debug it easily) and I have this problem.

I think it's a problem with an Eclipse debug launch, but user doesn't use it, so it is not an issue. We can close the issue.

Sorry for the noise and many thanks for your relevant feedback!

[cdietrich]

But it should work in this case too So the question is why it does not

[angelozerr]

As you said it's a problem with encoding, when my launch have UTF-8 it fails:

But when I swith to ISO-8859-1

It doesn' t fail but the renderer is bad:

[cdietrich]

I tested with utf8 and no issue Also it is default in lsp4j can you debug the client and check the exception

[angelozerr]

can you debug the client and check the exception

As soon as I will have time to debug LSP4E, I will give you feedback.