Closed tanishiking closed 1 year ago
Closing, this is just for saving / indexing this doc, for more discussion, it should go https://github.com/scalameta/metals-feature-requests/issues/236
Hey, is there any update/expected timeline on this? The feature would be awesome, I'd give anyone $50 who gets this done in the next couple of months.
No timeline yet, this requires quite a bit if work and currently we need to prioritize stability issues.
@tadeohepperle Surely you meant $50k?
@tadeohepperle Surely you meant $50k?
@alexarchambault Look, I just wanted to demonstrate that this is a valuable feature I'd be willing to pay for. If we had 1000 people interested in it like me, we'd have the $50k. I know $50 doesn't go anywhere on it's own. I'm not expecting any dev to get excited by $50...
Let's park the discussion about money here. As I said this is a non trivial matter and we don't plan on crowdfunding the efforts in the repo. Currently, we don't have the capacity nor time to work on it, but this might change in the future.
Another negative to using AlmondBSP that I want to point out @tgodzik (and this would be major issue for our setup), is that we want to have our local code available to the notebook (and keep our existing build tool). If you use AlmondBSP then you introduce a separate issue: how do you then export your non-notebook code/libs into Almond's build.
See more discussions from https://github.com/scalameta/metals/pull/4309
Language support for VSCode notebook
Overview
We can run Scala programs on Jupyter Notebooks using a Jupyter Kernel for Scala called Almond.
Currently, when using notebook cells in VSCode, Metals cannot provide language support such as code completion (by Metals) and code navigations.
This design document describes how Metals can support many language features for the notebook environment in VSCode
Scope
The scope of this document is limited to notebook in VSCode. It doesn't cover LSP support for jupyter notebook in general (like integration with jupyter-lsp).
However, supporting the notebooks in VSCode would be a good first step for integrations with other notebook environments.
Goal
textDocument/*
requests should be properly handled.textDocument/foldingRange
may be useless in notebook) though).Non-Goal
jupyter-lsp
).*.ipynb
file and no Scala Kernel is available.A.scala
adjacent toB.ipynb
,B.ipynb
should be able toimport
something fromA.scala
.Context
LSP 3.17 NotebookCellTextDocumentFilter
Starting from LSP 3.17, LSP clients can recognize notebook cells as documents and request
textDocument/*
to LSP servers. For example, the following change unlocks thetextDocument/*
requests for jupyter-notebook (of Scala cells) in VSCode.see: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#notebookCellTextDocumentFilter
Now, notebook on VSCode starts sending the
textDocument/*
request to LSP server.LSP 3.17 Notebook Document syncronization
In addition to
NotebookCellTextDocumentFilter
, LSP 3.17 provides a feature called Notebook Document syncronization.see: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#notebookDocument_synchronization
By setting server capabilities in
scalameta/metals
Notebook LSP client start sending several notifications such as
notebookDocument/didOpen
, andnotebookDocument/didChange
. LSP 3.17 expects LSP servers to syncronize the notebook contents by these notifications, and provide language features based on the synchronized content.Problems
Though LSP 3.17 provides some specifications around notebook support, it doesn't work out of the box with Metals for mainly two reasons.
Metals doesn't understand the Scala program in a cell
As you found, LSP client (VSCode) sends
textDocument/*
request withuri
points to the notebook cell.position
represents the position in the cell.However, Metals can't understand each cell of notebook. For example, in the following setting, Metals can't handle cell2 and cell3 (in Scala2, Metals can't cell1 too). Because each cell doesn't compile.
In order to analyze Scala code and provide compiler-based language features, Metals have to convert cells into compilable Scala program.
Metals server cannot recognise
*.ipynb
file as a compile targetMetals server provide language features to the files that are recognized as a
buildTarget/sources
of Build Server Protocol.However, current Metals doesn't recognize
*.ipynb
(or combined Scala source, that we'll mention later) for Build Servers.*Metals have to run build server that recognizes `.ipynb
files as a
SourceItem`**. Otherwise, Metals cannot tell which Scala version of presentation compiler should run against the notebook (Kernel).Solution / Technical Architecture
In order to provide language support for notebooks, two things are needed.
*.ipynb
as a build target.ScalaBuildTarget.scalaVersion
should match with the Scala version of the running Kernel.Components
Almond BSP
Almond Kernel
, but the I/O will be via unix domain socket at a pre-defined location (somewhere in~/.local/share/almond/bsp/{kernel-name}.sock
). See: https://github.com/scalameta/metals/pull/4253#issuecomment-1222091839buildTarget/compile
will convert cells into compilable Scala sources (as Almond Kernel / Ammonite does) and write the generated Scala sources onto the file system (as Ammonite Build Server does). Also,buildTarget/compile
should enable SemanticDB plugin.workspace/buildTargets
. One notebook - one buildTarget.buildTarget/scalacOptions
andScalacOptions.classDirectory
should point to the directory where Metals can find the Almond-generated Scala sources.workspace/wrappedSources
as ScalaCli doesVSCode
textDocument/*
requests toMetals
and receive responses.notebookDocument/didOpen|didChange
to Metals server.Metals
notebookDocument/didOpen
, connect to Almond BSP and register the result ofworkspace/buildTargets
.textDocument/*
request and convert the params to the generated Scala source. And then convert the result back to the cell positions.System flow
When VSCode start running Almond Kernel on VSCode
notebookDocument/didOpen
, then Metals will try to connect to the Almond BSP via the sock file created at (1)workspace/buildTargets
to Almond BSP, and Almmond BSP answers.When editing notebook
notebookDocument/didChange
.buildTarget/compile
to Almond BSP for generating the compilable Scala source files and generate SemanticDBs.buildTarget/compile
When VSCode request
textDocument/*
<->
generated Scala source for the cell, usingMappedSource
.<->
position in the generated Scala source can be done as the same way with Almond script.Alternative Solution
Can we just combine cells into Ammonite script, instead of communicating with Almond BSP?
While Almond can generate accurate Scala source program, it takes many steps for providing notebook support.
Instead of using Almond, Metals can combine cells into
*.sc
file.For example, given the following notebook cells,
Metals can combine those cells into
combined.sc
. The combine source will be a valid Ammmonite script file. Now Metals can re-use the Ammonite BSP for the language support.one notebook <-> one generated scala source
, that might be easier to handle (while Almond BSP willone notebook cell <-> one generated scala source
).cell <-> combined sc file <-> Ammonite generated scala file
.Run Almond BSP only when it's from VSCode
Instead of running Almond BSP all the time, maybe we can run Almond BSP only when it's from VSCode? I don't think we can, because:
Almond BSP shouldn't accept
buildTarget/compile
?Instead of accepting
buildTarget/compile
and generate Scala sources on notebook changes, we can let Kernel compile Scala sources (enabling SemanticDB and writing out the generated Scala sources to the filesystem).It works only if Almond Kernel can detect it's running for Metals, but it looks like impossible as mentioned in the above section.
Milestones
build/initialize
workspace/buildTargets
should returns the appropriate Scala version*.ipynb
withnotebookDocument/didOpen
, connect to Almond Build Server, and handshake.buildTarget/compile
buildTarget/scalacOptions
andScalacOptions.classDirectory
should point to the directory where Metals can find the Almond-generated Scala sources.buildTarget/compile
for every notebook changesnotebook cell URI <-> generated Scala source (found from ScalacOptions.classDirectory)
.textDocument/*
requests using presentation compiler.textDocument/*
requests using SemanticDBbuildTarget/compile
.Concerns
notebookDocument/didOpen
before Almond BSP starts (?)