Functional tests

[cdunklau]

~Work in progress~ well, it's still in progress, but it's functional enough to warrant some extra pairs of eyes.

functional-tests/README.md has some good details, probably start by reading that.

As of 7d6cc51, there are tests for:

• ban discovery from channel banlist
• noticing when a ban is un-set in a channel, notifying the unsetter with details
• auto-unset of expired bans
• redent command

To make the webui interaction not suck to test, I tweaked infobob's webui to output some things as JSON when given Content-Type: application/json. I also added a bit more logging in its IRCClient subclass.

[cdunklau]

The first idea I had for this would not be fun. See #56

[cdunklau]

in 8c432d8, there's an issue with pytest-twisted (I think?) where if I ctrl+c the test run after infobob connects, but before the deferLater fires, it throws illegal switch in blockon (twice, actually).

@altendky the README in functional-tests/ is at least halfway okay. If you're feeling up to it, follow the install setup first, then run the ircd/services with docker-compose, wait for atheme to say it uplinked (like m_pong(): finished synching with uplink (260 ms)), and then run tests-env/bin/pytest -s test_basic.py. Thanks in advance <3

Here's the output I get:

$ tests-env/bin/pytest -s test_basic.py 
/Users/cdunklau/Development/forks/infobob/functional-tests/tests-env/lib/python3.7/site-packages/twisted/internet/address.py:101: DeprecationWarning: The usage of `cmp` is deprecated and will be removed on or after 2021-06-01.  Please use `eq` and `order` instead.
  @attr.s(hash=False, repr=False, cmp=False)
============================= test session starts ==============================
platform darwin -- Python 3.7.5, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/cdunklau/Development/forks/infobob
plugins: twisted-1.12
collected 1 item                                                               

test_basic.py running ['/Users/cdunklau/Development/forks/infobob/functional-tests/infobob-env/bin/twistd', '-n', 'infobob', 'infobob.conf.json']
Infobob started
Got chunk of stdout: 2020-02-11T01:15:36+0100 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.4.0 (/Users/cdunklau/Development/forks/infobob/functional-tests/infobob-env/bin/python2 2.7.17) starting up.
2020-02-11T01:15:36+0100 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.selectreactor.SelectReactor.
Got chunk of stdout: 2020-02-11T01:15:36+0100 [-] Site starting on 8888
Got chunk of stdout: 2020-02-11T01:15:36+0100 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site instance at 0x10ecb5280>
Got chunk of stdout: 2020-02-11T01:15:36+0100 [infobob.irc.InfobobFactory#info] Starting factory <infobob.irc.InfobobFactory instance at 0x10cee6f00>
Got chunk of stdout: 2020-02-11T01:15:36+0100 [infobob.pastebin#info] Checking if pastebin u'bpaste' is up
Got chunk of stdout: 2020-02-11T01:15:36+0100 [infobob.pastebin#info] Attempting to retrieve 'https://bpaste.net'
Got chunk of stdout: 2020-02-11T01:15:36+0100 [twisted.web.client._HTTP11ClientFactory#info] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x10ed09140>
Got chunk of stdout: 2020-02-11T01:15:36+0100 [infobob.pastebin#info] Checking if pastebin u'habpaste' is up
Got chunk of stdout: 2020-02-11T01:15:36+0100 [twisted.web.xmlrpc._QueryFactory#info] Starting factory <twisted.web.xmlrpc._QueryFactory instance at 0x10ed12d70>
Got chunk of stdout: 2020-02-11T01:15:37+0100 [infobob.pastebin#critical] Unable to communicate with u'habpaste' pastebin

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionDone: Connection was closed cleanly.
Got chunk of stdout: 2020-02-11T01:15:37+0100 [infobob.pastebin#info] Recorded latency for pastebin u'habpaste' as 0.145106077194 seconds
Got chunk of stdout: 2020-02-11T01:15:37+0100 [twisted.web.xmlrpc._QueryFactory#info] Stopping factory <twisted.web.xmlrpc._QueryFactory instance at 0x10ed12d70>
Got chunk of stdout: 2020-02-11T01:15:37+0100 [infobob.pastebin#info] Recorded latency for pastebin u'bpaste' as 0.162493944168 seconds
^CGot chunk of stdout: 2020-02-11T01:15:39+0100 [-] Received SIGINT, shutting down.
FE

==================================== ERRORS ====================================
___________________ ERROR at teardown of test_infobob_basic ____________________

tmp_path = PosixPath('/private/var/folders/bf/_z_75dfd7cx9fky2x3vt8lq80000gp/T/pytest-of-cdunklau/pytest-94/test_infobob_basic0')

    @pytest.fixture(name='start_infobob')
    def fixture_start_infobob(tmp_path):
        infobob_python = pathlib.Path(os.environ['INFOBOB_PYTHON']).resolve()
        infobob_twistd = str(infobob_python.parent.joinpath('twistd'))
        called = False
        proctransport = None
        ended = None

        def start_infobob(channelsconf, autojoin):
            nonlocal called
            nonlocal proctransport
            nonlocal ended
            if called:
                raise RuntimeError('already called')
            called = True
            confpath = tmp_path.joinpath('infobob.conf.json')
            dbpath = tmp_path.joinpath('infobob.db')
            conf = {
                'irc': {
                    'server': SERVER,
                    'port': SERVER_PORT,
                    'ssl': False,
                    'nickname': INFOBOB_NICK,
                    'password': INFOBOB_PASS,
                    'nickserv_pw': None,
                    'autojoin': autojoin,
                },
                'channels': {
                    'defaults': {
                        'commands': [
                            ['allow', 'all'],
                        ],
                    },
                    **channelsconf,
                },
                'database': {'sqlite': {'db_file': dbpath.name}},
                'web': {'port': INFOBOB_WEB_PORT},
                'misc': {'manhole': {'socket': None}},
            }
            conn = sqlite3.connect(str(dbpath))
            with conn:
                conn.executescript(SCHEMA.read_text())
            conn.close()
            confpath.write_text(json.dumps(conf))
            from twisted.internet import reactor
            proto = InfobobProcessProtocol()
            args = [infobob_twistd, '-n', 'infobob', confpath.name]
            print('running', args)
            childFDs = {
                0: open(os.devnull, 'rb').fileno(),
                #1: open(os.devnull, 'wb').fileno(),
                1: 'r',
                #2: open(os.devnull, 'wb').fileno(),
                2: 'r',
            }
            ended = proto.ended
            proctransport = reactor.spawnProcess(
                proto,
                infobob_twistd,
                args=args,
                env=None,
                path=str(tmp_path),
                childFDs=childFDs,
            )

        yield start_infobob
        if proctransport is not None:
            proctransport.signalProcess('TERM')
>           return pytest_tw.blockon(ended)

test_basic.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests-env/lib/python3.7/site-packages/pytest_twisted.py:74: in blockon
    return blockon_default(d)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

d = <Deferred at 0x1093dd450>

    def blockon_default(d):
        current = greenlet.getcurrent()
        assert (
            current is not _instances.gr_twisted
        ), "blockon cannot be called from the twisted greenlet"
        result = []

        def cb(r):
            result.append(r)
            if greenlet.getcurrent() is not current:
                current.switch(result)

        d.addCallbacks(cb, cb)
        if not result:
            _result = _instances.gr_twisted.switch()
>           assert _result is result, "illegal switch in blockon"
E           AssertionError: illegal switch in blockon
E           assert () is []

tests-env/lib/python3.7/site-packages/pytest_twisted.py:92: AssertionError
=================================== FAILURES ===================================
______________________________ test_infobob_basic ______________________________

pyfuncitem = <Function test_infobob_basic>

    def pytest_pyfunc_call(pyfuncitem):
        if _instances.gr_twisted is not None:
            if _instances.gr_twisted.dead:
                raise RuntimeError("twisted reactor has stopped")

            def in_reactor(d, f, *args):
                return defer.maybeDeferred(f, *args).chainDeferred(d)

            d = defer.Deferred()
            _instances.reactor.callLater(
                0.0, in_reactor, d, _pytest_pyfunc_call, pyfuncitem
            )
>           blockon_default(d)

tests-env/lib/python3.7/site-packages/pytest_twisted.py:235: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

d = <Deferred at 0x10937e550 waiting on Deferred at 0x10937e690>

    def blockon_default(d):
        current = greenlet.getcurrent()
        assert (
            current is not _instances.gr_twisted
        ), "blockon cannot be called from the twisted greenlet"
        result = []

        def cb(r):
            result.append(r)
            if greenlet.getcurrent() is not current:
                current.switch(result)

        d.addCallbacks(cb, cb)
        if not result:
            _result = _instances.gr_twisted.switch()
>           assert _result is result, "illegal switch in blockon"
E           AssertionError: illegal switch in blockon
E           assert None is []

tests-env/lib/python3.7/site-packages/pytest_twisted.py:92: AssertionError
========================== 1 failed, 1 error in 3.97s ==========================

[cdunklau]

Looks like this was reported a while ago in pytest-dev/pytest-twisted#4

[cdunklau]

As of 0ed3cec the illegal switch in blockon thing still happens with ctrl+c, but I haven't been able to provoke it otherwise.