Parallel Testing with xdist

[benjamin-kirkbride]

I tried to use pytest-xdist to do parallel testing, but it didn't work. It seems like some state is shared between the running instances somehow. I'm not really sure what though.

Thoughts?

[Akuli]

I have tried it too, but I have forgotten if I ever figured out why it didn't work.

[Akuli]

I think it has to do with focusing. Some tests assume that you have the Porcupine window focused. But with xdist there are two Porcupine windows, and they can't both simultaneously have the keyboard focus.

[benjamin-kirkbride]

How does xvfb-run handle this? Is it possible that it's less about "having" focus than "having then losing" focus?

[Akuli]

Here is more evidence that it has to do with keyboard focus:

The sequence .F.F.F.F.FF. indicates to me that one of the two windows is consistently passing the tests and the other is failing. Also these tests use event_generate() which is a bit fragile and I think depends on the focus.

xvfb-run

I believe it emulates giving focus to the most recently opened window. That's probably why I get very consistently half of these event_generate() dependent tests failing also under xvfb.

[Akuli]

I just realized that xvfb doesn't run a window manager, so I think there's not really "multiple windows" to speak of. I'm not really sure what happens instead.

It is apparently possible to run a window manager inside xvfb: https://github.com/TestFX/TestFX/issues/158#issuecomment-62421691

[Akuli]

Lol. I think I got a window manager to work inside xvfb.

Without window manager, .focus_get() always returns None, maybe it means that focusing isn't really a thing or something. With xfwm4 running in the background it returns the root window itself because it apparently got focused. (No specific reason to use xfwm4 of all window managers, it's just what I happen to have)

[Akuli]

I just realized I can take screenshots inside xvfb-run + bash sessions by running scrot, a command-line tool that saves a screenshot as image file with no GUI interaction.

The window manager is definitely doing something. Without window manager, python3 -c 'import tkinter; tkinter.Tk().mainloop()' & looks like this:

With xfwm4 it looks like this:

I think my xvfb experiments don't help with this issue, but at least I learned more about how stuff works :)

[benjamin-kirkbride]

https://github.com/Akuli/porcupine/pull/1366 appears to help somewhat with this issue.

Here are two examples of me trying with -n auto:

FAILED tests/test_autocomplete_plugin.py::test_pasteId_lastPasteId - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_indent_dedent.py::test_indent_block_plugin - AssertionError: assert 'foo\nbar\nbiz\nbaz' == 'foo\n    bar\n    biz\nbaz'
FAILED tests/test_autocomplete_plugin.py::test_rare_thing_goes_last - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_autocomplete_plugin.py::test_case_sensitive_match_goes_first - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_indent_dedent.py::test_autoindent - AssertionError: assert '    if blah:  # comment' == '    if blah:...ent\n        '
FAILED tests/test_autocomplete_plugin.py::test_not_suggesting_what_you_just_typed - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_indent_dedent.py::test_shift_enter_and_alt_enter - AssertionError: assert False
FAILED tests/test_indent_dedent.py::test_dedent_on_closing_paren - AssertionError: assert 'print(\n    foo\n    ' == 'print(\n    foo\n)'
FAILED tests/test_indent_dedent.py::test_space_inside_braces_bug - AssertionError: assert '( aa ' == '( aa )'
FAILED tests/test_indent_dedent.py::test_double_dedent_bug - AssertionError: assert '        return foo' == '        return foo\n    '
FAILED tests/test_indent_dedent.py::test_dedent_blank_line_in_tabs_file_bug - AssertionError: assert '\tfoo\n\n\tbar' == 'foo\n\nbar'
FAILED tests/test_comment_block_plugin.py::test_comment_block_and_undo - AssertionError: assert 'foo\nbar\nbazlol' == '#foo\n#bar\n#bazlol'
FAILED tests/test_comment_block_plugin.py::test_partially_commented - AssertionError: assert 'We select st...as selected\n' == '#We select s...as selected\n'
FAILED tests/test_comment_block_plugin.py::test_cant_uncomment_bug - AssertionError: assert '    def __in...      raise\n' == '    def __in...      raise\n'
FAILED tests/test_indent_dedent.py::test_css_indented_as_if_it_was_python_bug - AssertionError: assert '\n' == 'b'
FAILED tests/test_default_keybindings.py::test_backspace_in_beginning_of_file[<BackSpace>] - AssertionError: assert 'a' == ''
FAILED tests/test_default_keybindings.py::test_backspace_in_beginning_of_file[<Control-Shift-BackSpace>] - AssertionError: assert 'a' == ''
FAILED tests/test_indent_dedent.py::test_markdown_autoindent - AssertionError: assert '1. Lol andwa...z.End of list' == '1. Lol and\n...\nEnd of list'
FAILED tests/test_indent_dedent.py::test_shell_autoindent - AssertionError: assert 'case foo inb...ho $thingdone' == 'case foo in\... $thing\ndone'
FAILED tests/test_indent_dedent.py::test_html_autoindent - assert '<!DOCTYPE ht.../body></html>' == '<!DOCTYPE ht...ody>\n</html>'
FAILED tests/test_indent_dedent.py::test_json_autoindent - assert '{"pygments_s...foo","bar"]}}' == '{\n  "pygmen...    ]\n  }\n}'

and

FAILED tests/test_autocomplete_plugin.py::test_pasteId_lastPasteId - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_indent_dedent.py::test_indent_block_plugin - AssertionError: assert 'foo\nbar\nbiz\nbaz' == 'foo\n    bar\n    biz\nbaz'
FAILED tests/test_autocomplete_plugin.py::test_rare_thing_goes_last - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_indent_dedent.py::test_autoindent - AssertionError: assert '    if blah:  # comment' == '    if blah:...ent\n        '
FAILED tests/test_autocomplete_plugin.py::test_case_sensitive_match_goes_first - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_find_plugin.py::test_key_bindings_that_are_annoying_if_they_dont_work - assert None is <porcupine.textutils.MainText object .!panedwindow.!panedwindow.!...
FAILED tests/test_indent_dedent.py::test_shift_enter_and_alt_enter - AssertionError: assert False
FAILED tests/test_autocomplete_plugin.py::test_not_suggesting_what_you_just_typed - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_indent_dedent.py::test_dedent_on_closing_paren - AssertionError: assert 'print(\n    foo\n    ' == 'print(\n    foo\n)'
FAILED tests/test_indent_dedent.py::test_space_inside_braces_bug - AssertionError: assert '( aa ' == '( aa )'
FAILED tests/test_indent_dedent.py::test_double_dedent_bug - AssertionError: assert '        return foo' == '        return foo\n    '
FAILED tests/test_indent_dedent.py::test_dedent_blank_line_in_tabs_file_bug - AssertionError: assert '\tfoo\n\n\tbar' == 'foo\n\nbar'
FAILED tests/test_comment_block_plugin.py::test_comment_block_and_undo - AssertionError: assert 'foo\nbar\nbazlol' == '#foo\n#bar\n#bazlol'
FAILED tests/test_comment_block_plugin.py::test_partially_commented - AssertionError: assert 'We select st...as selected\n' == '#We select s...as selected\n'
FAILED tests/test_indent_dedent.py::test_css_indented_as_if_it_was_python_bug - AssertionError: assert '\n' == 'b'
FAILED tests/test_comment_block_plugin.py::test_cant_uncomment_bug - AssertionError: assert '    def __in...      raise\n' == '    def __in...      raise\n'
FAILED tests/test_default_keybindings.py::test_backspace_in_beginning_of_file[<BackSpace>] - AssertionError: assert 'a' == ''
FAILED tests/test_default_keybindings.py::test_backspace_in_beginning_of_file[<Control-Shift-BackSpace>] - AssertionError: assert 'a' == ''
FAILED tests/test_indent_dedent.py::test_markdown_autoindent - AssertionError: assert '1. Lol andwa...z.End of list' == '1. Lol and\n...\nEnd of list'
FAILED tests/test_indent_dedent.py::test_shell_autoindent - AssertionError: assert 'case foo inb...ho $thingdone' == 'case foo in\... $thing\ndone'
FAILED tests/test_rstrip_plugin.py::test_rstrip - assert 'print("hello")  ' == 'print("hello")\n'
FAILED tests/test_tabs.py::test_read_only_file - ValueError: too many values to unpack (expected 1)
FAILED tests/test_indent_dedent.py::test_html_autoindent - assert '<!DOCTYPE ht.../body></html>' == '<!DOCTYPE ht...ody>\n</html>'
FAILED tests/test_indent_dedent.py::test_json_autoindent - assert '{"pygments_s...foo","bar"]}}' == '{\n  "pygmen...    ]\n  }\n}'

[Moosems]

Yikes, that's a lot of fails.