Features of sos languages

[BoPeng]

codemirror-sos.ts implements a CodeMirror 5 -style stream highlighter for sos. Since CodeMirror 6 uses a totally different mechanism, we need to write the language module using the new CodeMirror API (see an Example). The new implementation should use a Lezer Parser System. This Lexer guide provides a good starting point, and Grammer Nesting is the key technology we will need to use.

The new parser needs to implement the following features:

1. Magic (%MAGIC)

2. Cells with different kernels uses their native syntax highlighter mode (Python and R in the following example)

3. SoS mode is based on Python

4. In workflow mode, language switch to R, Python etc after action: specification

5. expand=True highlights expressions

6. Different expand delimiters

7. %expand magic works for SoS and non-SoS code cells

Relevant code:

1. parser

https://github.com/vatlab/jupyterlab-sos/blob/498aaa3d39d1a16e468226b1f98a6fbde3c34ce8/src/codemirror-sos.ts#L73-L75C29

2. base_mode should be an option for non-sos cells

https://github.com/vatlab/jupyterlab-sos/blob/498aaa3d39d1a16e468226b1f98a6fbde3c34ce8/src/codemirror-sos.ts#L195C1-L200

3. find mode for sos actions

https://github.com/vatlab/jupyterlab-sos/blob/498aaa3d39d1a16e468226b1f98a6fbde3c34ce8/src/codemirror-sos.ts#L470-L474

4. Parse %expand and determine left and right sigil

https://github.com/vatlab/jupyterlab-sos/blob/498aaa3d39d1a16e468226b1f98a6fbde3c34ce8/src/codemirror-sos.ts#L279C11-L295

These code snippets are just for reference. It looks like the new language module is based on parser instead of states, which should be much easier to implement.

[BoPeng]

Step 0 (done)

define a codemirror mode which is identical to Python, and use it in jupyterlab-sos.

[BoPeng]

import { pythonLanguage  } from "@codemirror/lang-python"
import { rLanguage } from "codemirror-lang-r"
import { markdownLanguage } from "@codemirror/lang-markdown"
import { javascriptLanguage} from "@codemirror/lang-javascript"

import { StreamLanguage } from "@codemirror/language"
import { ruby } from "@codemirror/legacy-modes/mode/ruby"
import { sas } from "@codemirror/legacy-modes/mode/sas"
import { octave } from "@codemirror/legacy-modes/mode/octave"
import { perl } from "@codemirror/legacy-modes/mode/perl"
import { shell } from "@codemirror/legacy-modes/mode/shell"
import { EditorView } from "@codemirror/view"
import { EditorSelection } from "@codemirror/state"

const defaulAction: ActionLang[] = [
  {
    action: "Python",
    parser: pythonLanguage.parser
  },
  {
    action: "R",
    parser: rLanguage.parser
  },
  {
    action: "Javascript",
    parser: javascriptLanguage.parser
  },
  {
    action: "matlab",
    parser: StreamLanguage.define(octave).parser
  },
  {
    action: "octave",
    parser: StreamLanguage.define(octave).parser
  },
  {
    action: "ruby",
    parser: StreamLanguage.define(ruby).parser
  },
  {
    action: "markdown",
    parser: markdownLanguage.parser
  },
  {
    action: "perl",
    parser: StreamLanguage.define(perl).parser
  },
  {
    action: "shell",
    parser: StreamLanguage.define(shell).parser
  },
  {
    action: "run",
    parser: StreamLanguage.define(shell).parser
  },
  {
    action: "sh",
    parser: StreamLanguage.define(shell).parser
  },
  {
    action: "bash",
    parser: StreamLanguage.define(shell).parser
  },
  {
    action: "sas",
    parser: StreamLanguage.define(sas).parser
  }
]

[BoPeng]

Step 1:

In jupyterlab-sos allow

    (cell.inputArea.editorWidget.editor as CodeMirrorEditor).setOption(
      'mode',
      'sos'
    );

It is uncertain

1. If codemirror side allows options such as parseConf
2. How to change parseConf dynamically from the jupyterlab side.

[BoPeng]

Step 2

Add support for

1. %magic
2. [step ]
3. Action:

to the base Python mode.

[BoPeng]

Step 3:

Add support for sigil and expansion for Python mode.

[BoPeng]

Step 4:

Add support for language option to choose python/r etc as basemode, and use base mode

[BoPeng]

step 5:

Add support %expand to base model

[BoPeng]

Question asked on https://discourse.jupyter.org/t/how-to-set-codemirror-mode-for-individual-editor/22026