[Code structure] Indentation error messes up highlighting

[microbit-carlos]

Bug Description

When an indentation error is added to the code the code structure highlighting is not following the rest of the code correctly.

How To Reproduce

# Add your Python code here. E.g.
from microbit import *

a = True
while a:
     display.scroll('micro:bit')
    display.show(Image.HEART)
    while a:
        display.scroll('micro:bit')
        while a:
            display.scroll('micro:bit')
            display.show(Image.HEART)
            sleep(2000)
        display.show(Image.HEART)
        sleep(2000)
    sleep(2000)

Steps to reproduce the behavior:

1. Copy the code into the editor
2. See error

Expected behavior

Code structure highlighting to correctly follow the code indentation

Screenshots

Environment

• OS: macOS 11
• Browser: Chrome
• Version: v98

Additional context

N/A

[microbit-matt-hillsdon]

The problem is that the dedent ends the "while" so the parse for the second while has it at top level.

The lines we draw are based on the AST and an idealised 4-space indentation rather than the actual text position. We could revisit this and draw them based on whitespace.

We could also use a subset of Python with strict 4 space block indent at the grammar level to simplify error this kind of scenario (painful changes in Pyright and CM's parser but logically sensible).

One for discussion.

[microbit-matt-hillsdon]

This is working as intended at the moment, though there are other options we could consider based on further feedback.

I think to match Carlos's expectations you'd need to stop using the AST and just become indentation based, though we'd then show highlighting/lines for invalid code.