Reorganize client source with lerna and typescript projects

To easing proposing the client-facing parts of this codebase to @jupyterlab/jupyterlab, the repository could be re-organized as a monorepo with smaller, independently testable, extensible, and replaceable components with well-defined inter-package interfaces.

This would not block any feature development, but should help keep PRs more focused and less likely to create conflicts.

step 0️⃣#77

get src compiling with composite: true
- remove all requires

step 1️⃣ #79

move src into packages/jupyterlap-lsp
add tsconfigbase.json to be used by sub-package tsconfig.jsons
create new private lerna package.json
- owns common activities, like lint and build
create metapackage as a composite of the other packages
- this is the only place tsc will be invoked
change build to use the metapackage
otherwise no substantive code changes

step 2️⃣

break up jupyterlab-lsp into language-server and language-server-extension
- these will need new names at this point
- language-server
- add a tokens.ts and an ILanguageServerManager
- centralize CommandIds
identify opportunities for lazy-loading to reduce impact on vendor-main with await import
- the extension will end up being in main, and just about everything from the manager will likely be lazy-loadable, e.g. the first time a notebook/document is loaded. on subsequent loads, it will already be cached.

step 3️⃣

break out features, as relevant, into smaller sub-packages, adopting Manager registration APIs
- especially per-language or per-activity features, e.g. language-server-ipython, language-server-notebook
- this will make it more obvious how to contribute new language/features
- this also paves the way to contribution by extension, e.g. gutters and codeactions from @deathbeds/lintotype, hierarchical symbols (perhaps as part of @jupyterlab/toc)
we can also bring in adopted upstreams, such as jump-to-definition and the parts of lsp-codemirror

remember when i said i would be better about doing more manageable chunks of work? well...

I started taking a look at what it would take to add a hierarchical symbol viewer, and it got a little more... involved than I thought... basically a good chunk of [2] and [3]:

https://github.com/krassowski/jupyterlab-lsp/compare/master...bollwyvl:add-hierarchial-symbol-viewer

here's the yak shave, each one of which should probably be a pr:

add a new package boilerplate, but can't talk to lsp
- add a token on lsp for the new package to require
- refactor how the connection manager work so that it could allow adding a capabilit
  - bring in lsp-ws to start hacking on it
  - redo the build to accomodate the need for webpacking ws because net
    - upgrade typescript
      - upgrade lab
      - find some actual issues in ws while using optional chaining from ts 3.7
- get back to an actually working lab (just got here, probably done for the night)
- [x] let symbol viewer change the init params
- [x] add api to register for new messages
- [x] draw the symbols in a tree (first pass, probably just <details>, but probably pick up some react nonsense
- [ ] allow jumping from the symbol to the source doc

So anyhooo... I'll keep going on it, but it would already be unreasonably large to review, breaks tests, etc. so i'll probably have to actually make all those prs above.

however, in trying to get a new end-to-end feature actually working, i'm starting to see the pattern of some relatively radical things we could do to make it fit in the lab architecture:

one websocket per language
- one init message
- all the ws methods accept a vscode-uri
replace events with phosphor (ne lumino) messages or signals
replace a lot of hardcoded strings with direct references to the vscode-jsonrpc
- we're shipping it anyway
lots of lazy loading

The features refactor brought as closer to step 3. Useful notes are in the comment https://github.com/krassowski/jupyterlab-lsp/issues/316#issuecomment-673527034.

The plan in top level comment is slightly outdated, but I will work to make it happen; the next steps after #738 are:

step 3: split features into standalone packages. I am thinking about -feature suffix so that would be @jupyter-lsp/diagnostics-feature, and it would allow us to have @jupyter-lsp/diagnostics in case if we wanted to make some low-level shared code available to third-parties (as @jupyterlab does with -extension)
- we could go for more technical -capability, but that would actually go to the non-suffixed one...
step 2: split jupyterlab-lsp into packages as needed to avoid circular import in step 3; we should not call it "language-server" as it is client; currently I'm thinking:
- @jupyter-lsp/jupyterlab-lsp - umbrella package for compatibility
- @jupyter-lsp/ui-components - today's components/
- @jupyter-lsp/virtual - virtual document, editor and friends
- @jupyter-lsp/protocol - today's lsp.ts; this needs to be available to features and provides what upstream package does not (e.g. fixes to upstream bugs like this one) and enums available at runtime
- @jupyter-lsp/editor-integration
- @jupyter-lsp/codemirror4-integration
- @jupyter-lsp/adapter
- @jupyter-lsp/notebook-adapter
- @jupyter-lsp/file-adapter
- @jupyter-lsp/core - not sure of name for this one, but basically should provide IFeature, FeatureSettings, positions, and all other things which are needed for features and not worth splitting into individual packages

Probably not this year though...

Update thoughts:

adapters don't need to be in three separate packages initially.

Edit: The split-up is also useful since starting with Notebook v7 it will possibly serve not only JupyterLab but also Notebook as demonstrated on RetroLab example. For now we could have

@jupyterlab/lsp - provides core services starting with Notebook v7 and Lab v4
@jupyter-lsp/core - core services required by features (for now including functionality moved to @jupyterlab/lsp)
@jupyter-lsp/signature-capability, @jupyter-lsp/hover-capability, etc
@jupyter-lsp/ui-components
@jupyter-lsp/ipython-extractors
@jupyter-lsp/rpy2-extractors
@jupyter-lsp/jupyterlab-lsp - bundle of capabilities for Lab and later @jupyter-lsp/notebook-lsp - bundle of capabilities for Notebook (some capabilities may not be as useful in the notebook interface and could be excluded, and some widget positioning code could be specific to Notebook app)

jupyter-lsp / jupyterlab-lsp