Problems understanding PipeInput

[fspegni]

Hi. I'm studying your library since a while, but didn't find a proper way of fixing my issue.

In brief: I have a customized app, eventloop, and cli. I want the user to pass a custom input (let's say a PipeInput) to the cli.

I send 4 \n separated lines on the PipeInput, but it looks to me the prompt doesn't process three commands.

A minimal script is:

import prompt_toolkit as pt

def my_prompt(prompt_in):
    """
    Inspired by prompt_toolkit.shortcuts.prompt()
    """

    line = None

    try:
        app = pt.shortcuts.create_prompt_application(
            erase_when_done=True,
        )
        el = pt.shortcuts.create_eventloop()
        cli = pt.interface.CommandLineInterface(
            application=app,
            input=prompt_in,
            eventloop=el,
            output=pt.shortcuts.create_output(true_color=False)
        )
        patch_context = pt.utils.DummyContext()

        with patch_context:
            doc = cli.run(reset_current_buffer=True)

            if doc:
                line = doc.text.strip()

    except EOFError:
        line = None

    return line

# setup the input
prompt_in = pt.input.PipeInput()
prompt_in.send("foo\nfie\nfoo\nfie\n")

# start the interpretation loop

while True:

    try:

        line = my_prompt(prompt_in)

        if line is None:
            break

        print "[%s]" % line
    except KeyboardInterrupt:
        print "Goodbye"
        break
    except Exception, e:
        print "Error: %s" % e
        break

The output is:

user@host:wd$ python example.py  
foo
[foo]
^[[23;1R^[[23;9R

I can't understand what those strange sequences of characters mean, and how to debug this problem.

Hope you can shed some light :) Thanks

[fspegni]

I forgot to specify: as you can see the interpretation stops at the first command (foo), instead of continuing with the next

[jonathanslenders]

Hi @fspegni,

PipeInput is actually mostly meant for unit-testing. You could use it to redirect input, for instance in the case of an SSH or Telnet connection, where no TTY was allocated, but usually, you'd read from stdin. What is the actual use case you're trying to implement?

Regarding the escape characters that you see. They are the CPR response. Cursor-position-request-response. In order to know how much space there is available below the prompt, we send a CPR request to stdout. The terminal will reply with the cursor coordinates through stdin. You would have to read these characters from stdin.

But again, could you explain me the scenario you're trying to implement? (Where is the input coming from?)

Jonathan

[fspegni]

Hi @jonathanslenders, actually the example I reported was extracted from a more complex use case. In my use case I want to take the input either from stdin or from a file (specifying a flag in my python program). In case of an input file, I extended the class prompt_toolkit as follows:

class PromptInput(pt.input.Input):

    def __init__(self, script_path=None):

        self.buffer = ""
        self.is_interactive = True

        try:
            if script_path:
                self.f_in = open(script_path, "r")
                self.is_interactive = False
            else:
                self.f_in = sys.stdin

            # test the file has returns correctly from fileno()
            self.f_in.fileno()

        except Exception, e:
            self.f_in = sys.stdin
            raise plugins.CommandAbort("Cannot read script file '%s': %s" % (script_path, e))

    def fileno(self):
        return self.f_in.fileno()

    def read(self):

        if len(self.buffer) == 0:
            self.buffer = self.f_in.read()

        # if there is a newline, returns the
        # buffer up to that position
        pos = self.buffer.find("\n")

        if pos < 0:
            pos = len(self.buffer)

        read_string = self.buffer[:pos]

        # delete the returned data from buffer
        self.buffer = self.buffer[pos:]

        return read_string

    def raw_mode(self):

        if self.is_interactive:
            mode = raw_mode(self.f_in.fileno())
        else:
            mode = pt.utils.DummyContext()
        return mode

    def cooked_mode(self):

        if self.is_interactive:
            mode = cooked_mode(self.f_in.fileno())
        else:
            mode = pt.utils.DummyContext()
        return mode

It is still an on going work, so let me know if you find any flaws in it. The first thing I noticed is that the read(...) method is not called by your library. Is there a reason why the Input class should provide such a method, then?

The second issue I had to fixed, as you mentioned, was to get rid of those CPR responses that I don't need, when reading from a file. Thus, I replaces the create_output(...) invocation with a DummyOutput() instance. Though, I find the name create_output a bit misleading, because it does not mention that it assumes it creates a terminal output.

So far no change in your code was needed. At the end, though, I had problems because your library was reading as much as possible (up to 1024 bytes) at a time, not caring whether in the read data a newline \n character was present (see the read(...) method in eventloop/posix_utils.py). This is a problem when reading from a file (but not when reading from the stdin). To avoid it, at the moment I forced the code to read 1 byte at at time (not the best choice, I know), replacing:

            data = os.read(self.stdin_fd, count)

with:

            data = os.read(self.stdin_fd, 1)

As you said, we should probably address this as a new case study and fix the architecture of the library, if you think it's worth incorporating it. I think the library is very well done, but this use case is quite common, so it's worth adding it. I can help providing some pull requests that you can supervise, in particular:

• for taking an input parameter in your shortcuts functions, so that one does not have to write a method similar to my my_prompt method, just to specify a custom (extension of) Input
• introduce a FileInput extension of Input
• not sure what to do with the output: there is no need for a FileOutput class, for this use case, because the output remains the stdout that is a terminal, but apparently this case study should avoid sending CPR requests
• fixing the os.read(...) invocation to read count bytes, but returning only up to the first \n in case the multiline flag is not set to True and keepting the rest of the read data in a buffer that will be returned the next invocation of the read method defined in eventloop/posix_utils.py
• it's not clear to me what to do with the read(...) method required by the Input class: does it play any role?

I'm available for helping and discussing this use case. Thanks for all your work.

[fspegni]

Any news on this?