Wrong indentation after Python 3.6 f-string with literal string

[dwoschee]

Recent versions of jupyter notebook highlight the part in curly braces of an f-string as normal code and not as a string. However, if that code part contains a literal string, the quote that closes the f-string is interpreted as a quote starting a new string, and the end of the f-string is not found for highlighting.

This has two consequences:

1. The code in the rest of the line is highlighted incorrectly. For example, if the f-string is an argument of a function, the closing bracket is shown in red like a string.

2. If the f-string is inside brackets (function call, list definition etc.), all new lines in the cell are automatically indented to the beginning of the brackets containing the f-string, as if the content of the brackets was continued in a new line.

The following screenshot shows some examples: All cells are evaluated as expected; only highlighting is broken.

While the first consequence is merely a graphical issue, the second consequence interferes with auto-indentation and heavily affects the user experience since the indentation has to be fixed manually for each new line inserted below the f-string.

I’m using a fresh venv environment with pip as package manager. Here is some system information:

sys.version:
        3.6.6 (default, Jul 19 2018, 16:29:00)
        [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]

platform.platform():
        Linux-4.17.14-102.fc27.x86_64-x86_64-with-fedora-27-Twenty_Seven

pip list:
        Package            Version
        ------------------ -------
        backcall           0.1.0
        bleach             3.0.2
        decorator          4.3.0
        defusedxml         0.5.0
        entrypoints        0.2.3
        ipykernel          5.1.0
        ipython            7.0.1
        ipython-genutils   0.2.0
        ipywidgets         7.4.2
        jedi               0.13.1
        Jinja2             2.10
        jsonschema         2.6.0
        jupyter            1.0.0
        jupyter-client     5.2.3
        jupyter-console    6.0.0
        jupyter-core       4.4.0
        MarkupSafe         1.0
        mistune            0.8.4
        nbconvert          5.4.0
        nbformat           4.4.0
        notebook           5.7.0
        pandocfilters      1.4.2
        parso              0.3.1
        pexpect            4.6.0
        pickleshare        0.7.5
        pip                18.1
        prometheus-client  0.4.1
        prompt-toolkit     2.0.6
        ptyprocess         0.6.0
        Pygments           2.2.0
        python-dateutil    2.7.3
        pyzmq              17.1.2
        qtconsole          4.4.1
        Send2Trash         1.5.0
        setuptools         37.0.0
        simplegeneric      0.8.1
        six                1.11.0
        terminado          0.8.1
        testpath           0.4.2
        tornado            5.1.1
        traitlets          4.3.2
        wcwidth            0.1.7
        webencodings       0.5.1
        widgetsnbextension 3.4.2

[minrk]

This is likely a codemirror issue, and may require updating the codemirror version in the notebook package.