IHaskell support

[jamesdbrock]

I'm trying to use the ghcide LSP server for Haskell with this extension, with these Language Server User Preferences:

{                                     
  "language_servers": {                    
    "ghcide-language-server": {            
      "version": 1,                        
      "argv": ["/opt/bin/ghcide", "--lsp", "--verbose"],
      "languages": ["Haskell"]             
    }                                      
  }
}

After the message LSP: Untitled1.ipynb ready for connection, an error is thrown from this line:

https://github.com/krassowski/jupyterlab-lsp/blob/f305ab4e1d8ff6b62daffe1460636175406a00df/packages/jupyterlab-lsp/src/adapters/jupyterlab/jl_adapter.ts#L212

this is of type NotebookAdapter, and this.mime_type is undefined, and the error says:

react_devtools_backend.js:2273 TypeError: Cannot read property 'split' of undefined
   at NotebookAdapter.get language [as language] (jl_adapter.js:126)

[krassowski]

We had a similar issue with Scala kernel if I remember correctly and the problem was on the Scala side (i.e. the kernel was not returning mime type correctly)...

[krassowski]

Whoops, it was Robot kernel, not Scala. See https://github.com/robots-from-jupyter/robotkernel/issues/40.

[krassowski]

The actual mime-type is derived from the kernel language info and fallbacks to cell metadata if none, see:

https://github.com/krassowski/jupyterlab-lsp/blob/f305ab4e1d8ff6b62daffe1460636175406a00df/packages/jupyterlab-lsp/src/adapters/jupyterlab/notebook.ts#L116-L123

For file editor, the mime-type provided by the JupyterLab mime-type registry should be used:

https://github.com/krassowski/jupyterlab-lsp/blob/f305ab4e1d8ff6b62daffe1460636175406a00df/packages/jupyterlab-lsp/src/adapters/jupyterlab/file_editor.ts#L25-L27

[krassowski]

By the way, thank you for reporting it! Please give us an update on how it is going, and please feel welcome to contribute your spec.

[jamesdbrock]

Oh yeah thanks @krassowski , it looks like you're right, the IHaskell kernel is not reporting mime_type as required here.

https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-info

https://github.com/gibiansky/IHaskell/blob/cc427517a59bfc8e1014c7e02fe5cfe0ce7e5eef/ipython-kernel/src/IHaskell/IPython/Types.hs#L294-L301

[bollwyvl]

On our side, we could also try and back it out of codemirror_mode, which we could trace back to haskell mode. More workarounds relying on implementation-specific information lead to suffering (as we've learned on this project)... no doubt we'll be seeing kernels specifically targeting the vscode notebook behavior, and they might not include codemirror_mode.

So, in the meantime, we've probably done a Good Thing in highlighting this on multiple kernels, as it's a really good piece of standards-compliant(ish) metadata... but given it's come up twice, we should probably be more inclusive.

[bollwyvl]

And, i'll echo @krassowski 's thanks in posting this!

Near term: if you get it working, we would love to get your recipe added to the docs, or link out to a gist, etc.

Longer term: official support for haskell would be great!

Our gates for inclusion thus far has been something like (roughly in order):

• works on binder
  • this is a great way to demonstrate MVP: can just be a gist
  • there are a ton of examples out there
• added to docs
• can be installed with 1-3 new files to check in, like a list of dependencies and a bootstrap script
  • this should cover:
  • the language runtime server
  • the language runtime (if not in a "convenience" language like nodejs, but seems highly unlikely for haskell!)
  • at least one linter (if separate)
  • in this case, the kernel!
  • if all on conda-forge, that's a big plus...
  • or we can investigate getting them packaged for conda-forge, if it makes any sense at all
• has some robot test coverage (e.g. an example file with syntax errors that get linted)
• can be tested in CI on linux (at a minimum)
  • and osx (better)
  • and windows (best)
  • and this is usually awful, and why we try to use conda
• populates as much of the spec as is reasonable

This does the most for the "Greater Good," even though it would end up requiring maintenance, as it would help keep us honest moving forward for features important to you so that the next person doesn't have to struggle up the same learning curve... or just quit!

[jamesdbrock]

I patched the mimetype issue in IHaskell, that's fixed.

Now my next problem is that I have anther runtime null execption because the LanguageServerManager._sessions has no entry for "haskell". (It does have an entry for "python" → "pyls" because I also installed python-language-server. )

https://github.com/krassowski/jupyterlab-lsp/blob/d7ac678975f65b920f54b3034c9bbddd978d98bd/packages/jupyterlab-lsp/src/connection_manager.ts#L325-L327

Is my Language Server User Preferences in the first post of this issue not properly registering with the LanguageServerManager?

[krassowski]

It just occurred to me that you appear to set the specs in the wrong place; The use of language_servers suggests that you are going for settings editor, which is not where the specs are located. I think that this is our fault because we call it "Configuration" in the docs, but really it could be called "Adding LSP server" to distinguish it from the settings (i.e. the configuration sent to the language server once it is up an running).

Please see Configuring. You basically need to create a python spec file as seen here: https://github.com/krassowski/jupyterlab-lsp/tree/master/py_src/jupyter_lsp/specs (for example pyls.py) and put it in a folder which is discoverable by jupyter (use jupyter --paths as described in the docs).

@bollwyvl would JSON-based specs still work? I think we do not use any in the repo.

[bollwyvl]

Yes, the JSON specs still work, have used them for some test implementations... here's the mimimum one I did for testing out LanguageServer.jl:

{
  "LanguageServerManager": {
    "language_servers": {
      "julia-languageserver": {
        "version": 3,
        "argv": ["julia", "--debug=yes", "--project=.", "-e", "using LanguageServer, LanguageServer.SymbolServer; runserver()", "."],
        "languages": ["julia"],
        "display_name": "LanguageServer.jl",
        "mime_types": ["text/julia", "text/x-julia"]
      }
    }
  }
}

Note: this is from an experimental branch: the current schema version is 2... but the docs say 1: we need to fix and test the docs...

The quick test of this is putting it in jupyter_notebook_config.json in the same directory where the notebook server is started.

For packaging, etc. this can be put into:

$CONFIG/jupyter_notebook_config.d/languageserverjl.json

where $CONFIG is any of the well-known locations config from:

jupyter --paths --json

[jamesdbrock]

Thanks for the advice guys. I'm currently trying this:

jovyan@08874484ae98:~$ jupyter --paths
config:
    /home/jovyan/.jupyter
    /opt/conda/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
    ...

jovyan@08874484ae98:~$ cat /etc/jupyter/jupyter_notebook_config.d/ghcide-language-server.json 
{
  "LanguageServerManager": {
    "language_servers": {
      "ghcide-language-server": {
        "version": 2,
        "argv": ["/opt/bin/ghcide", "--lsp"],
        "languages": ["haskell"],
        "display_name": "ghcide",
        "mimetypes": ["text/haskell", "text/x-haskell"]
      }
    }
  }
}

"haskell" is still not getting registered in LanguageServerManager and results in runtime null exception...

[bollwyvl]

Thanks for keeping with it!

For simplicty:

• close lab browser
• stop lab server
• clear console
• jupyter troubleshoot
• jupyter labextension list
• jupyter notebook extension
• put that config in ./jupyter_notebook_config.json
• fire up lab with jupyter lab --debug
• open a haskell file
• open a haskell notebook
• post the notebook and browser log (scrubbing, i guess) as a gist

Some potential gotchas embedded in that:

• as "closest" to the user, config in ./jupyter_notebook_config.json will always win
• all of the process-level config has to be baked in basically at "install" time
  • we've got another approach in the works (#268 moving the stuff in the notebook server to a dedicated proxy client, which would be dynamic) but for now,
• if it works with files, but not the notebook it's one class of problem, as there is a lot if python/r-specific handling for that format, as those are what current maintainers/contributors have cared to ensure works well

[jamesdbrock]

Okay my configuration is working as ./jupyter_notebook_config.json

[jamesdbrock]

[jamesdbrock]

As you guessed @bollwyvl , .hs files seem to work but Haskell notebooks not so much.

[krassowski]

Well, it seems that it works in the notebook too, however, the language server reads it as if it was a file (as expected - this it is how the LSP for notebooks work - there is no concept of a cell nor of a notebook in the LSP - the cells are merged together and the server is given a flat file, which we call "virtual document"), thus requires an entrypoint (I based my Hasklell knowledge on this answer).

We workaround kernel-specific additions to the syntax that deviate from valid files using regular expressions to modify the resulting "virtual document" (see positioning system illustration which demonstrates that we can split a single notebook into multiple virtual documents, and replace the Python-kernel specific %% magics with valid code for each language). Thes replacements are defined in magics module.

I imagine that maybe it would help if we could prepend each virtual document with main = (this would have no effect on the interactive execution whatsoever - we would just be tricking the LSP server to believe that it is a valid .ts file). Do you think that it would be enough to make it work?

[jamesdbrock]

maybe it would help if we could prepend each virtual document with main =

Yeah, I see what you're saying @krassowski . What you're describing wouldn't work. I can't think of a simple transformation from code cells to virtual document that would work.

There are nine different kinds of things which can be in an IHaskell code cell:

https://github.com/gibiansky/IHaskell/blob/49b03cf5a9a8e8f38a617551b80acf081f4ecc14/src/IHaskell/Eval/Parser.hs#L38-L48

It will be tricky to make this work.

[krassowski]

What kind of transformation would need to be done to support IHaskell then? Btw we could also treat each cell as an independent virtual document. Or, is there any working LSP IHaskell integration for notebook out there?

[jamesdbrock]

I tucked my configuration into the branch https://github.com/jamesdbrock/ihaskell-notebook/tree/lsp while we give this a think.

[jamesdbrock]

Thanks @krassowski and @bollwyvl , This is a really nice JupyterLab extension, I hope we can get it working for IHaskell.