FAIL: test_no_spawn_no_stdin_attached (tests.commands.test_global.TestExternalCommand)

[GuillaumeSeren]

Hello, Thank you for alot, my favorite MUA :rocket:

The issue append when I run the test-suite (in the gentoo package), here the error log:

======================================================================                                                                                                              
FAIL: test_no_spawn_no_stdin_attached (tests.commands.test_global.TestExternalCommand)
----------------------------------------------------------------------
Traceback (most recent call last):      
  File "/var/tmp/portage/mail-client/alot-9999/work/alot-9999/tests/utilities.py", line 189, in _actual
    return loop.run_until_complete(coro(*args, **kwargs))                     
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()                                                                                                                                                          
  File "/var/tmp/portage/mail-client/alot-9999/work/alot-9999/tests/commands/test_global.py", line 129, in test_no_spawn_no_stdin_attached
    ui.notify.assert_not_called()                                                                                                                                                   
  File "/usr/lib/python3.7/unittest/mock.py", line 840, in assert_not_called                                                                                                        
    raise AssertionError(msg)                                                             
AssertionError: Expected 'notify' to not have been called. Called 1 times.

----------------------------------------------------------------------

The issue append on 0.9 / 0.9.1 / and master.

Software Versions

• python3.7
• notmuch-0.29.3-r2
• alot-0.9.1

To Reproduce The issue append also in userspace, when running pytest, see

└─> pytest
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-5.4.2, py-1.8.0, pluggy-0.13.1
rootdir: /home/gseren/src/free/github/guillaumeseren/alot
plugins: hypothesis-4.57.1
collected 336 items                                                                                                                                                                

===================================================================================== FAILURES =====================================================================================
_______________________________________________________________ TestExternalCommand.test_no_spawn_no_stdin_attached ________________________________________________________________

self = <tests.commands.test_global.TestExternalCommand testMethod=test_no_spawn_no_stdin_attached>

    @utilities.async_test
    async def test_no_spawn_no_stdin_attached(self):
        ui = utilities.make_ui()
        cmd = g_commands.ExternalCommand('test -t 0', refocus=False)
        await cmd.apply(ui)
>       ui.notify.assert_not_called()

tests/commands/test_global.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

_mock_self = <Mock name='mock.notify' id='140597591781584'>

    def assert_not_called(_mock_self):
        """assert that the mock was never called.
        """
        self = _mock_self
        if self.call_count != 0:
            msg = ("Expected '%s' to not have been called. Called %s times." %
                   (self._mock_name or 'mock', self.call_count))
>           raise AssertionError(msg)
E           AssertionError: Expected 'notify' to not have been called. Called 1 times.

/usr/lib/python3.7/unittest/mock.py:840: AssertionError
============================================================================= short test summary info ==============================================================================
FAILED tests/commands/test_global.py::TestExternalCommand::test_no_spawn_no_stdin_attached - AssertionError: Expected 'notify' to not have been called. Called 1 times.
===================================================================== 1 failed, 326 passed, 9 xfailed in 5.51s =====================================================================

But userspace works with the following:

$ python3.7 setup.py test 
...
test_no_spawn_failure (tests.commands.test_global.TestExternalCommand) ... ok                                                                                                       
test_no_spawn_no_stdin_attached (tests.commands.test_global.TestExternalCommand) ... ok                                                                                             test_no_spawn_no_stdin_success (tests.commands.test_global.TestExternalCommand) ... ok                                                                                              
test_no_spawn_stdin_attached (tests.commands.test_global.TestExternalCommand) ... ok                                                                                                test_no_spawn_stdin_success (tests.commands.test_global.TestExternalCommand) ... ok                                                                                                 
test_spawn_failure (tests.commands.test_global.TestExternalCommand) ... ok   
...
Ran 346 tests in 3.575s                                                                                                                                                             
                                                                                                                                                                                    OK (expected failures=9) 

It is the only test failing, maybe it is one the dependencies:

dev-python/sphinx-2.4.4
app-crypt/gpgme-1.13.1
dev-python/configobj-5.0.6
dev-python/python-magic-0.4.15-r1
dev-python/urwid-2.1.0
dev-python/urwidtrees-1.0.2-r1
dev-python/twisted-20.3.0
net-mail/mailbase-1.5-r1 
net-mail/notmuch-0.29.3-r2
dev-python/mock-3.0.5-r1
dev-lang/python-3.6.10-r2 
dev-lang/python-3.7.7-r2
dev-lang/python-exec-2.4.6-r1
dev-python/setuptools-44.1.0 

[pazz]

As far as I understand, pytest is a third party module and setuptools (and by extension, alot) uses the unittest module for tests. So, python3.7 setup.py test is the correct way to run our test suite and pytest is not. Perhaps pytest has some conflict with the mock library unittest.mock?

In any case, I cannot reproduce this on my machine (python 3.8.2) nor does this happen on Travis, even for python 3.7 as far as I can see. So I'm not sure how to help you here..

[GuillaumeSeren]

Hey @pazz thank you for the reply.

As far as I understand, pytest is a third party module and setuptools (and by extension, alot) uses the unittest module for tests.

Yes my explanation was a bit confuse, but I used unittest in the package, and got this issue, in my package building env, like

FAIL: test_no_spawn_no_stdin_attached (tests.commands.test_global.TestExternalCommand)
----------------------------------------------------------------------
Traceback (most recent call last):                                                       
  File "/var/tmp/portage/mail-client/alot-0.9.1/work/alot-0.9.1/tests/utilities.py", line 189, in _actual
    return loop.run_until_complete(coro(*args, **kwargs))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/var/tmp/portage/mail-client/alot-0.9.1/work/alot-0.9.1/tests/commands/test_global.py", line 129, in test_no_spawn_no_stdin_attached
    ui.notify.assert_not_called()                                                                                                                                                  
  File "/usr/lib/python3.7/unittest/mock.py", line 840, in assert_not_called                                                                                                       
    raise AssertionError(msg)                                                                                                                                                      
AssertionError: Expected 'notify' to not have been called. Called 1 times.     

----------------------------------------------------------------------
Ran 345 tests in 10.530s                                                                 

FAILED (failures=1, expected failures=9

While testing in the userspace I discovered a that I can reproduce the issue on test_no_spawn_no_stdin_attached from the package building env.

_______________________________________________________________ TestExternalCommand.test_no_spawn_no_stdin_attached _______________________________________________________________

self = <tests.commands.test_global.TestExternalCommand testMethod=test_no_spawn_no_stdin_attached>

    @utilities.async_test
    async def test_no_spawn_no_stdin_attached(self):
        ui = utilities.make_ui()
        cmd = g_commands.ExternalCommand('test -t 0', refocus=False)
        await cmd.apply(ui)
>       ui.notify.assert_not_called()

tests/commands/test_global.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_mock_self = <Mock name='mock.notify' id='140589940215696'>

    def assert_not_called(_mock_self):
        """assert that the mock was never called.
        """
        self = _mock_self
        if self.call_count != 0:
            msg = ("Expected '%s' to not have been called. Called %s times." %
                   (self._mock_name or 'mock', self.call_count))
>           raise AssertionError(msg)
E           AssertionError: Expected 'notify' to not have been called. Called 1 times.

/usr/lib/python3.7/unittest/mock.py:840: AssertionError

Running the testsuite, 'setup.py test' works whithout errors.

Maybe there is some permission mechanism similar in pytest and the package building env, I am investigating and will report here.

[GuillaumeSeren]

After investingating a bit more, I think the origin is the sandbox mechanism in the test phase, so I am keeping this test disabled for now.