Transformation.display_to_source is never called

[plotski]

When I'm looking at the code, the only call to display_to_source is made in BufferControl.mouse_handler. If mouse_support=False, display_to_source is never called. If mouse_support=True, display_to_source is only called for clicks.

Here is some demo code:

from prompt_toolkit import Application
from prompt_toolkit.buffer import Buffer
from prompt_toolkit.key_binding import KeyBindings
from prompt_toolkit.layout.containers import Window
from prompt_toolkit.layout.controls import BufferControl
from prompt_toolkit.layout.layout import Layout
from prompt_toolkit.layout.processors import Processor, Transformation

class MyProcessor(Processor):
    def apply_transformation(self, transformation_input):
        return Transformation(
            fragments=transformation_input.fragments,

            # This exits the application as expected.
            #source_to_display=exit,

            # This does nothing unless mouse support is enabled and the
            # BufferControl is clicked.
            display_to_source=exit,
        )

root_container = Window(BufferControl(
    buffer=Buffer(),
    input_processors=[MyProcessor()],
))

kb = KeyBindings()
@kb.add('c-q')
def exit_(event):
    event.app.exit()

app = Application(
    layout=Layout(root_container),
    key_bindings=kb,
    mouse_support=True,
)
app.run()

The application runs normally until you click on it.

Setting mouse_support=False turns display_to_source into dead code.

Uncommenting source_to_display=exit confirms that the Transformation object is used.

Is this a bug or am I misunderstanding how this is supposed to work?

[plotski]

I've tried really hard to fix this issue over the weekend but it seems impossible.

I want to have structured user input, like a form but with all fields in a single line. The structure comes from a simple markup language I made up, something like <first name>Marquis</first name> <last name>de Sade</last name>.

I subclassed Processor with an apply_transformation() method that removes the markup. This works nicely. The user sees "Marquis de Sade" with no markup while my application can parse the markup and clearly distinguish between first name and last name. The cursor is even placed correctly thanks to Transformation.source_to_display().

But cursor movement doesn't work because the Buffer doesn't know anything about Processors and "move right" is applied to the markup, not to the displayed text. The user moves the cursor but it doesn't actually move if it is somewhere on the hidden markup.

I've attached another demo at the end. To keep it simple, it's hiding the character "l" from the user.

How to fix this?

Somehow, Buffer or Document (which one?) needs to get access to the input_processors passed to BufferControl. But it can't call apply_transformation() without width and height, which are not available during cursor movement.

The BufferControl instance holds _ProcessedLine instances (which should re-use the previously rendered width and height) which I can get by calling _last_get_processed_line. I've tried attaching _last_get_processed_line to the Buffer instance, but that didn't really go anywhere.

In theory, this could be used to get the displayed text, apply any movement operations on that, and then apply _ProcessedLine.display_to_source() to the resulting cursor position. In practice, I have no idea how to implement this. And I don't think it's that simple for inserting/deleting input because I can't perform the editing on the displayed text and then convert it back to markup.

This whole approach feels like a dead end and I'd be grateful for any help.

from prompt_toolkit import Application
from prompt_toolkit.buffer import Buffer
from prompt_toolkit.key_binding import KeyBindings
from prompt_toolkit.layout.containers import HSplit, Window
from prompt_toolkit.layout.controls import BufferControl, FormattedTextControl
from prompt_toolkit.layout.layout import Layout
from prompt_toolkit.layout.processors import Processor, Transformation

class HiddenLProcessor(Processor):
    def apply_transformation(self, transformation_input):
        fragments = transformation_input.fragments
        source_text = ''.join(frag[1] for frag in fragments)
        display_text = ''.join(c for c in source_text if c != 'l')

        def source_to_display(source_index):
            # Translate cursor position in source to cursor position in text.
            display_index = source_index
            for c in source_text[:source_index]:
                if c == 'l':
                    display_index -= 1
            return display_index

        def display_to_source(display_index):
            # Translate cursor position in text to cursor position in source.
            source_index = 0
            text_curpos = 0
            for c in source_text:
                if text_curpos >= display_index:
                    break
                source_index += 1
                if c != 'l':
                    text_curpos += 1
            return source_index

        # Show visually where the cursor is in each version of the text.
        def curpos(string, curpos):
            return string[:curpos] + '|' + string[curpos:]

        pos = transformation_input.document.cursor_position
        displayed_pos = source_to_display(pos)
        source_pos = display_to_source(displayed_pos)
        info.text = (
            f'Source cursor position: {pos:>3d}: {curpos(source_text, pos)!r}\n'
            f'Displayed cursor position: {displayed_pos:>3d}: {curpos(display_text, displayed_pos)!r}\n'
            f'Cursor position avoiding source code: {source_pos:>3d}: {curpos(source_text, source_pos)!r}\n'
        )

        return Transformation(
            fragments=[('', display_text)],
            source_to_display=source_to_display,
            display_to_source=display_to_source,
        )

info = Buffer()
input = Buffer()
input.text = 'Hello, World!'

kb = KeyBindings()
@kb.add('c-q')
def exit_(event):
    event.app.exit()

app = Application(
    layout=Layout(HSplit([
        Window(
            content=BufferControl(buffer=input, input_processors=[HiddenLProcessor()]),
            dont_extend_height=True,
        ),
        Window(content=BufferControl(buffer=info), dont_extend_height=True),
    ])),
    key_bindings=kb,
    mouse_support=True,
)
app.run()