LanguageServer can connect and provide basic functionality to NeoVim client

[scztt]

Based on some preliminary investigation, NeoVim can only connect to LSP servers via a stdin/stdout based communication mechanism. This currently will not work with sclang, as there are many places where sclang forcibly writes to stdout, which would disrupt LSP communication. Instead, for the vscode plugin, we use simple UDP pipes since they are already well-supported by sclang and easy to implement in VSCode. Until the sclang stdout issues can be solved, the UDP mechanism will be used.

IF it is indeed not possible to convince NeoVim to connect to an LSP server via UDP, then the solution to this would be as follows. (Note that this is not throwaway work, as an architecture where the LSP "wrapper" and an sclang instance are separated is probably desirable in the long run anyway.)

There is a preliminary implementation that may be a good starting point here: https://github.com/davidgranstrom/sclang-lsp-stdio

• [ ] Create a simple Python (or Lua, or nodejs) command line tool that can launch sclang. Command line tool should set correct environment variables (https://github.com/scztt/vscode-supercollider/blob/develop/src/context.ts#L49), and launch sclang with the correct arguments (https://github.com/scztt/vscode-supercollider/blob/develop/src/context.ts#L66).
• [ ] Connect to sclang's stdin/stdout pipes so that they can be rerouted or logged rather than e.g. forwarded to out command line scripts stdout.
• [ ] Connect command line script to sclang via UDP sockets (see https://github.com/scztt/vscode-supercollider/blob/develop/src/context.ts#L117). Connection should be made only after ***LSP READY*** is posted (https://github.com/scztt/vscode-supercollider/blob/develop/src/context.ts#LL132C45-L132C61). UDP sockets should be connected directly to stdin/stdout of the utility script so they can communicate with the LSP client (e.g. NeoVim).
• [ ] Lifetime and exit code of command line script should mirror that of sclang, e.g. if sclang exits script should also (with the same error code).

Note that there MAY be extra work in NeoVim specifically to enable SuperCollider support. There are forum messages outlining some of this work, and some things may already be implemented. See:

• https://scsynth.org/t/supercollider-vscode-language-server-protocol-support/5574/4
• https://scsynth.org/t/supercollider-vscode-language-server-protocol-support/5574/17
• https://scsynth.org/t/supercollider-vscode-language-server-protocol-support/5574/23

[ranjithshegde]

For information, Neovim 0.9 (or perhaps even earlier version) provides a method to connect to LSP rpc via tcp in function vim.lsp.rpc.connect

So something in these lines is already possible

local function start_lsp()
    vim.loop.spawn('/path/to/sclang', {args = {'-i', 'vscode'},}, function ()
       -- Some shutdown mechanism here 
    end)
end

vim.lsp.start({
  name="supercollider",
  cmd=function(...)
    start_lsp()
    return vim.lsp.rpc.connect(host, port)
  end
})

[dannyZyg]

Is the ideal here to write a program that results in a binary executable (or is python/node/lua fine)? There is a note in David's original proof of concept that suggests it would be good to not have a dependency on Node.

There are many LSPs which are installed via npm etc. so maybe node or whatever is fine.

I'd be happy to give this one a go if there is nobody working on it yet.

[dannyZyg]

I've had some time and have been working on a Python script to handle this. I've got most of the functionality in place but I've been testing outside of SC mostly.

When I try to run SC from my python script I get this (permission?) error.

The command I run on Mac is /Applications/SuperCollider.app/Contents/MacOS/sclang -l ~/Library/Application\ Support/SuperCollider/sclang_conf.yaml

I had to point it to my config file so it would pick up the Quarks.

Below is the error I get but strangely if I just run from the shell it works fine and I see the ***LSP READY*** message.

I'm using SCLANG_LSP_ENABLE=1.

Does anyone know what might be going on? Thanks!

compiling class library...
        Found 859 primitives.
        Compiling directory '/Applications/SuperCollider.app/Contents/Resources/SCClassLibrary'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Singleton'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Collapse'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/WindowViewRecall'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Log'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/UnitTest2'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Deferred'
        Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/LanguageServer'
        numentries = 877100 / 13676400 = 0.064
        5800 method selectors, 2358 classes
        method table size 14477520 bytes, big table size 109411200
        Number of Symbols 13017
        Byte Code Size 397053
        compiled 368 files in 0.37 seconds
compile done
ERROR: Primitive '_FileMkDir' failed.
caught exception 'boost::filesystem::create_directory: Read-only file system: "/synthdefs"' in primitive in method Meta_File:mkdir
RECEIVER:
class File (0x128c7e700) {
  instance variables [19]
    name : Symbol 'File'
    nextclass : instance of Meta_FileDialog (0x1287aa880, size=19, set=5)
    superclass : Symbol 'UnixFILE'
    subclasses : nil
    methods : instance of Array (0x128c7e880, size=12, set=4)
    instVarNames : instance of SymbolArray (0x128c7ea00, size=1, set=2)
    classVarNames : instance of SymbolArray (0x128c7eb80, size=1, set=2)
    iprototype : instance of Array (0x128c7eac0, size=1, set=2)
    cprototype : instance of Array (0x128c7ec40, size=1, set=2)
    constNames : nil
    constValues : nil
    instanceFormat : Integer 0
    instanceFlags : Integer 0
    classIndex : Integer 1119
    classFlags : Integer 0
    maxSubclassIndex : Integer 1119
    filenameSymbol : Symbol '/Applications/SuperCollider.app/Contents/Resources/SCClassLibrary/Common/Files/File.sc'
    charPos : Integer 0
    classVarIndex : Integer 220
}
CALL STACK:
        MethodError:reportError
                arg this = <instance of PrimitiveFailedError>
        Nil:handleError
                arg this = nil
                arg error = <instance of PrimitiveFailedError>
        Thread:handleError
                arg this = <instance of Thread>
                arg error = <instance of PrimitiveFailedError>
        Object:throw
                arg this = <instance of PrimitiveFailedError>
        Object:primitiveFailed
                arg this = <instance of Meta_File>
        String:mkdir
                arg this = "/synthdefs/"
        Meta_SynthDef:initClass
                arg this = <instance of Meta_SynthDef>
        Meta_Class:initClassTree
                arg this = <instance of Meta_Class>
                arg aClass = <instance of Meta_SynthDef>
                var implementsInitClass = nil
        ArrayedCollection:do
                arg this = [*217]
                arg function = <instance of Function>
                var i = 5
        Meta_Class:initClassTree
                arg this = <instance of Meta_Class>
                arg aClass = <instance of Meta_Object>
                var implementsInitClass = nil
        Process:startup
                arg this = <instance of Main>
                var time = 0.37496
        Main:startup
                arg this = <instance of Main>
                var didWarnOverwrite = false
^^ The preceding error dump is for ERROR: Primitive '_FileMkDir' failed.
caught exception 'boost::filesystem::create_directory: Read-only file system: "/synthdefs"' in primitive in method Meta_File:mkdir
RECEIVER: File

[jamshark70]

SynthDef initClass does:

    *initClass {
        synthDefDir = Platform.userAppSupportDir ++ "/synthdefs/";
        // Ensure exists:
        synthDefDir.mkdir;
    }

If synthDefDir == "/synthdefs/", then it means Platform.userAppSupportDir must be returning an empty string.

userAppSupportDir is populated in C++, in SC_Filesystem_macos.cpp. So the code in this file must not be compatible with the way neovim is launching sclang.

[dannyZyg]

Thanks for info, I was able to track down the issue by looking at that SC_Filesystem_macos.cpp file. I was setting some env vars for my subprocess but was overwriting my default env so was missing things like HOME.

Thanks!

[dannyZyg]

I've almost got this working - I think i need to work on my neovim lsp setup because currently when I open a SC file in neovim it will launch my lsp runner but it won't show a connected LSP.

It is definitely launching though and producing promising logs

───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ DEBUG:asyncio:Using selector: KqueueSelector
   2   │ INFO:__main__:RUNNER: SC env vars: {'SCLANG_LSP_ENABLE': '1', 'SCLANG_LSP_CLIENTPORT': '57210', 'SCLANG_LSP_SERVERPORT': '57211', 'SCLANG_LSP_LOGLEVEL': 'info'}
   3   │ INFO:__main__:RUNNER: Launching SC with cmd: ['/Applications/SuperCollider.app/Contents/MacOS/sclang', '-i', 'external', '-l', '/Users/danny/Library/Application Support/SuperCollider/sclang_conf.yaml']
   4   │ INFO:__main__:SC:STDOUT: compiling class library...
   5   │ INFO:__main__:SC:STDOUT:    Found 859 primitives.
   6   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Applications/SuperCollider.app/Contents/Resources/SCClassLibrary'
   7   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/.local/share/SuperCollider/Extensions'
   8   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Singleton'
   9   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Collapse'
  10   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/WindowViewRecall'
  11   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Log'
  12   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/UnitTest2'
  13   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/Deferred'
  14   │ INFO:__main__:SC:STDOUT:    Compiling directory '/Users/danny/Library/Application Support/SuperCollider/downloaded-quarks/LanguageServer'
  15   │ INFO:__main__:SC:STDOUT:    numentries = 877100 / 13676400 = 0.064
  16   │ INFO:__main__:SC:STDOUT:    5800 method selectors, 2358 classes
  17   │ INFO:__main__:SC:STDOUT:    method table size 14477520 bytes, big table size 109411200
  18   │ INFO:__main__:SC:STDOUT:    Number of Symbols 13017
  19   │ INFO:__main__:SC:STDOUT:    Byte Code Size 397053
  20   │ INFO:__main__:SC:STDOUT:    compiled 368 files in 0.32 seconds
  21   │ INFO:__main__:SC:STDOUT: compile done
  22   │ INFO:__main__:SC:STDOUT: localhost : setting clientID to 0.
  23   │ INFO:__main__:SC:STDOUT: internal : setting clientID to 0.
  24   │ INFO:__main__:SC:STDOUT: Class tree inited in 0.01 seconds
  25   │ INFO:__main__:SC:STDOUT: [LANGUAGESERVER.QUARK] Starting language server, inPort: 57210 outPort:57211
  26   │ INFO:__main__:SC:STDOUT: [LANGUAGESERVER.QUARK] Adding provider for method 'initialize'
  27   │ INFO:__main__:SC:STDOUT: ***LSP READY***
  28   │ INFO:__main__:RUNNER: ready message received
  29   │ INFO:__main__:SC:STDOUT: *** Welcome to SuperCollider 3.14.0-dev. *** For help type cmd-d.
  30   │ INFO:__main__:UDP SERVER: connection made
  31   │ INFO:__main__:UDP SERVER: connection made
  32   │ INFO:__main__:RUNNER: UDP Sender running on 127.0.0.1:57210
  33   │ INFO:__main__:SC:STDOUT: [LANGUAGESERVER.QUARK] Message received: 0.435571125, a NetAddr(127.0.0.1, 57845), Content-Length: 3645

strangely when i exit the neovim buffer I get this (which almost looks like an init call)

  34   │ INFO:__main__:SC:STDOUT: [LANGUAGESERVER.QUARK] Message received: 6.327646292, a NetAddr(127.0.0.1, 57845), {"params":{"rootUri":"file:\/\/\/Users\/danny\/Music","workspaceFolders":[{"name":"\/Users\/danny\/Music","uri":"file:\/\/\/Users\/
       │ danny\/Music"}],"initializationOptions":{},"processId":39910,"clientInfo":{"version":"0.9.1","name":"Neovim"},"trace":"off","capabilities":{"textDocument":{"callHierarchy":{"dynamicRegistration":false},"declaration":{"linkSupport":true},"r
       │ eferences":{"dynamicRegistration":false},"documentHighlight":{"dynamicRegistration":false},"documentSymbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]},"hierarchicalDocumentSymbolSuppor
       │ t":true,"dynamicRegistration":false},"signatureHelp":{"signatureInformation":{"parameterInformation":{"labelOffsetSupport":true},"documentationFormat":["markdown","plaintext"],"activeParameterSupport":true},"dynamicRegistration":false},"im
       │ plementation":{"linkSupport":true},"synchronization":{"willSave":true,"didSave":true,"willSaveWaitUntil":true,"dynamicRegistration":false},"publishDiagnostics":{"tagSupport":{"valueSet":[1,2]},"relatedInformation":true},"codeAction":{"code
       │ ActionLiteralSupport":{"codeActionKind":{"valueSet":["","quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"resolveSupport":{"properties":["edit"]},"isPreferredSupport":true,"
       │ dataSupport":true,"dynamicRegistration":false},"rename":{"prepareSupport":true,"dynamicRegistration":false},"typeDefinition":{"linkSupport":true},"definition":{"linkSupport":true},"completion":{"contextSupport":true,"completionItem":{"snip
       │ petSupport":true,"commitCharactersSupport":true,"preselectSupport":true,"deprecatedSupport":true,"documentationFormat":["markdown","plaintext"],"insertReplaceSupport":true,"insertTextModeSupport":{"valueSet":[1,2]},"labelDetailsSupport":tr
       │ ue,"resolveSupport":{"properties":["documentation","detail","additionalTextEdits","sortText","filterText","insertText","textEdit","insertTextFormat","insertTextMode"]},"tagSupport":{"valueSet":[1]}},"insertTextMode":1,"completionList":{"it
       │ emDefaults":["commitCharacters","editRange","insertTextFormat","insertTextMode","data"]},"completionItemKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]},"dynamicRegistration":false},"semanticTokens":{"
       │ tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator","de
       │ corator"],"formats":["relative"],"overlappingTokenSupport":true,"multilineTokenSupport":false,"serverCancelSupport":false,"augmentsSyntaxTokens":true,"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract",
       │ "async","modification","documentation","defaultLibrary"],"requests":{"range":false,"full":{"delta":true}},"dynamicRegistration":false},"hover":{"contentFormat":["markdown","plaintext"],"dynamicRegistration":false}},"workspace":{"workspaceF
       │ olders":true,"applyEdit":true,"workspaceEdit":{"resourceOperations":["rename","create","delete"]},"symbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]},"hierarchicalWorkspaceSymbolSuppor
       │ t":true,"dynamicRegistration":false},"semanticTokens":{"refreshSupport":true},"didChangeWatchedFiles":{"relativePatternSupport":true,"dynamicRegistration":false},"configuration":true},"window":{"workDoneProgress":true,"showMessage":{"messa
       │ geActionItem":{"additionalPropertiesSupport":false}},"showDocument":{"support":true}}},"rootPath":"\/Users\/danny\/Music"},"id":1,"method":"initialize","jsonrpc":"2.0"}
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[themissingcow]

@dannyZyg did you get any further with this - I should have some time to help soon. I would love to get it all working in neovim.

[dannyZyg]

@dannyZyg did you get any further with this - I should have some time to help soon. I would love to get it all working in neovim.

Hey @themissingcow - I haven't looked at this since last year, but I'd love to get it going if possible. I'll try and have a look in the next few days and let you know how I go. I can't quite remember where I got to with it.

[themissingcow]

@dannyZyg did you get any further with this - I should have some time to help soon. I would love to get it all working in neovim.

Hey @themissingcow - I haven't looked at this since last year, but I'd love to get it going if possible. I'll try and have a look in the next few days and let you know how I go. I can't quite remember where I got to with it.

🙏

Ace, thanks. I grabbed that branch on your fork - but had issues manually calling vim.lsp.start with the python script - it would spin up the command, so the output.log file appeared, and ready the LSP, but nvim would never attach it to the buffer... The inner workings of LSPs in nvim is new to me, so going to start from the top to try and understand it a little more.

[themissingcow]

we use simple UDP pipes since they are already well-supported by sclang

@scztt please forgive the naive q - has there been any chat upstream about adding tcp support to sclang?

[themissingcow]

@dannyZyg I managed to get this working in nvim, it needed a few tweaks to LSP.sc and the runner script. I'll try to package these up soon once I've tidied them up a little. Many thanks to you and @scztt for all the hard work on getting it this far - this is going to be so helpful!

[dannyZyg]

That's great @themissingcow ! Really glad to hear you carried on with it and fixed it. Curious to see what changes were needed. Excited to use this too!

[themissingcow]

That's great @themissingcow ! Really glad to hear you carried on with it and fixed it. Curious to see what changes were needed. Excited to use this too!

WIP: https://github.com/themissingcow/LanguageServer.quark/commits/nvim-fixes

Still having some issues where the last packet of multi-packet messages is getting lost. Should hopefully have some time next week to look into it again, and will post the updated Python wrapper.

[themissingcow]

@dannyZyg got bit further here https://github.com/themissingcow/LanguageServer.quark/commits/lsp-runner. Were you planning on working on the wrapper script some more? Otherwise I was thinking about adding support for configuring paths/config and the likes via LSP initialization options, so you can set these up in your neovim registration (and potentially per-project). Didn't want to mess with your plans though!

[dannyZyg]

Hey @themissingcow, thanks for continuing with this! I was planning on coming back to it at some point but had not got around to it, so I'm really pleased you picked it up. Your config suggestions sound good, I'd be happy for you to add that. It would be great to get all of this merged in though, so we can make small improvements as we go and have something workable soon! Let me know how you'd like to proceed with that?

What state is it in now? I'll try and test out your changes tonight!

[themissingcow]

Hey @dannyZyg

Your config suggestions sound good, I'd be happy for you to add that. It would be great to get all of this merged in though, so we can make small improvements as we go and have something workable soon! Let me know how you'd like to proceed with that?

Yeah def. I guess I was hoping to get the setting from the LSP initialise working as a starting point, but I guess it's kind of working as is.

What state is it in now? I'll try and test out your changes tonight!

I was just testing and cross-referenced the LSP docs and I realised that some of the changes I made in LSP.sc aren't actually needed. Ill update the branch to remove those (sorry I'll force-push vs having some revert commits, as we'd def not want to merge it upstream with those in).

[dannyZyg]

Yeah def. I guess I was hoping to get the setting from the LSP initialise working as a starting point, but I guess it's kind of working as is.

@themissingcow That's no problem, feel free to add that in! I don't mean to rush you, I'm just excited to use this haha 😄

[themissingcow]

@themissingcow That's no problem, feel free to add that in! I don't mean to rush you, I'm just excited to use this haha 😄

@dannyZyg I updated the branch, I have it kind of working (though I think there are still some gremlins with large messages).

I'm adding this to my lspconfig.configs:

local configs = require('lspconfig.configs')

 configs.supercollider = {
   default_config = {
     cmd = {"python", "/path/to/lsp_runner/main.py", "--sc-lang-path", '/Applications/SuperCollider.app/Contents/MacOS/sclang', "--sc-config-path", "/path/to/sclang_conf.yaml"},
     filetypes = {'supercollider'},
     root_dir = function(fname)
        return "/"
     end,
     settings = {},
   },
 }

And setting this up with require("lspconfig.configs").supercollider.setup({}), for some reason I had to override the builtin filetype detection as it was flagging my .sc files as Scala despite the logic in filetypes.lua...

vim.filetype.add({
  extension = {
    sc = "supercollider",
    scd = "supercollider",
  },
})

Which gives me this loveliness:

[dannyZyg]

Hi @scztt I've just put up a PR for this here. Thanks for the detailed outline here. Ready for you to take a look whenever you get the chance 🙏 thanks!