Testing questionary flows

[xmatthias]

It would be great if questionary would have an easy way to test input flows

For example (reusing my example flow from #34):

from questionary import Separator, test_prompt 
questions = [
        {
            "type": "confirm",
            "name": "conditional_step",
            "message": "Would you like the next question?",
            "default": True,
        },
       {
            "type": "text",
            "name": "next_question",
            "message": "Name this library?",
            "when": lambda x: x['conditional_step'],
            "validate": lambda val: val == "questionary"
        },
       {
            "type": "select",
            "name": "second_question",
            "message": "Select item",
            "choices": [
                "item1",
                "item2",
                Separator(),
                "other",
            ],
        },
        {
            "type": "text",
            "name": "second_question",
            "message": "Insert free text",
            "when": lambda x: x["second_question"] == "other"
        },
]
inputs = ["Yes", "questionary", "other", "free text something"]
vals = test_prompt(questions, inputs)

assert vals['second_question'] == "free text something"
. . .

Now by calling test_prompt() with an input string (or List - which is probably easier to compile) we can run through the whole workflow, and verify that all keys are populated as expected. This would allow proper CI for more complex flows ... which base one input on top of the other as in the question-flow above. I suspect it would be possible by mocking some internals of questionary - but i see this as a dangerous approach as every minor change in these mocked functions would probably lead to an error in my tests.

Most of the code / logic should already be available as part of the tests for questionary - however that's not available when installing from pypi...

[tmbo]

That is a really good idea, and actually something we have been struggling with in https://github.com/rasahq/rasa as well.

I need to digg a little deeper, but ideally, there should be a way to mock the actual IO, but provided by questionary (as you said, I don't think it is save for the user to mock these kind of things, but it would be fine to provide & test it as part of the library).

Could be something like

from questionary.test import mocked_input
# ...

def test_input_questions():
  with mocked_input({"conditional_step": "Yes", "next_question": "questionary"}):
    vals = prompt(questions)
  # ...

[yajo]

I can see that the test suite has some interesting utils to test itself:

https://github.com/tmbo/questionary/blob/1a2c99bd0b3998bd04714ae1fd12fd26bd4b8464/tests/utils.py#L12-L77

How about bundling all those utils in questionary and documenting them a little bit, so downstream projects can test too?

Notice that some good answer to this might land in https://github.com/prompt-toolkit/python-prompt-toolkit/issues/1243.

[yajo]

I've been using pexpect successfully in https://github.com/copier-org/copier/pull/260 to test workflow. I recommend it, although it won't work on Windows as explained in https://github.com/prompt-toolkit/python-prompt-toolkit/issues/1243#issuecomment-706668723.

[lyz-code]

I've gathered a small example using @Yajo 's method here in case anyone is as lost as I was some hours ago.

[SteffenBrinckmann]

The arrow-keys are such a great feature of questionary. Did anybody get the arrow-keys working for their testing environment? And if yes, could you post the code?

[kiancross]

The arrow-keys are such a great feature of questionary. Did anybody get the arrow-keys working for their testing environment? And if yes, could you post the code?

Some of our own tests use the arrow keys. You should be able to get this working in your own code by copying tests/utils.py. An example using arrow keys is here:

https://github.com/tmbo/questionary/blob/2979fb8c6404f69112e8ebf918f9a2b29a310b45/tests/prompts/test_select.py#L53

As @Yajo has suggested, we should expose these in the Questionary API so that people can more easily test their projects!

[pmeier]

Any update on this? Currently we are vendoring the test/utils.py, but it would be much nicer to have them in questionary.test or the like.

[kiancross]

@pmeier No update at the moment, but happy to provide feedback on any PRs which attempt to implement this.

[pmeier]

I'll clean up our test code to see what exactly we need and send a PR afterwards.