Two tests failing

[jluttine]

I'm running the test suite with pytest and two tests fail. Any ideas what could be wrong:

============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/nix-build-python2.7-txtorcon-0.19.3.drv-0/txtorcon-0.19.3, inifile:
collected 521 items

test/test_addrmap.py ..........
test/test_attacher.py ....
test/test_circuit.py ......................
test/test_controller.py .....................................................
test/test_endpoints.py ..............................................
test/test_fsm.py .........
test/test_log.py .
test/test_microdesc.py ....
test/test_router.py ................
test/test_socks.py ....................................
test/test_stream.py ........................
test/test_torconfig.py .............................................................................................
test/test_torcontrolprotocol.py ................................................................................
test/test_torinfo.py ................
test/test_torstate.py .............................................................
test/test_util.py ...F.........F........................
test/test_util_imports.py .
test/test_web.py .......

=================================== FAILURES ===================================
______________ TestGeoIpDatabaseLoading.test_return_geoip_object _______________

self = <test.test_util.TestGeoIpDatabaseLoading testMethod=test_return_geoip_object>

    @skipIf('pypy' in sys.version.lower(), "No GeoIP in PyPy")
    def test_return_geoip_object(self):
        from txtorcon import util
        (fd, f) = tempfile.mkstemp()
        ret_val = util.create_geoip(f)
        delete_file_or_tree(f)
>       self.assertEqual(type(ret_val).__name__, 'GeoIP')

test/test_util.py:76: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/nix/store/hbhww15hxsgs0y03gcc20p8ags148w5w-python2.7-Twisted-17.5.0/lib/python2.7/site-packages/twisted/trial/_synctest.py:432: in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
E   FailTest: 'NoneType' != 'GeoIP'
______________________ TestProcessFromUtil.test_real_addr ______________________

result = <<class 'twisted.internet.tcp.Port'> of test.test_util.FakeProtocolFactory (not listening)>
g = <generator object test_real_addr at 0x7fffef99d820>
deferred = <Deferred at 0x7fffef83af80 current result: None>

    def _inlineCallbacks(result, g, deferred):
        """
        See L{inlineCallbacks}.
        """
        # This function is complicated by the need to prevent unbounded recursion
        # arising from repeatedly yielding immediately ready deferreds.  This while
        # loop and the waiting variable solve that by manually unfolding the
        # recursion.

        waiting = [True, # waiting for result?
                   None] # result

        while 1:
            try:
                # Send the last result back as the result of the yield expression.
                isFailure = isinstance(result, failure.Failure)
                if isFailure:
                    result = result.throwExceptionIntoGenerator(g)
                else:
>                   result = g.send(result)

/nix/store/hbhww15hxsgs0y03gcc20p8ags148w5w-python2.7-Twisted-17.5.0/lib/python2.7/site-packages/twisted/internet/defer.py:1386: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test/test_util.py:225: in test_real_addr
    pid = process_from_address('0.0.0.0', port, self.fakestate)
txtorcon/util.py:218: in process_from_address
    stdout=subprocess.PIPE)
/nix/store/2nsq3cpqy514n5raqdzpfh6lprpqlmjk-python-2.7.13/lib/python2.7/subprocess.py:390: in __init__
    errread, errwrite)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x7ffff01ddf10>
args = ['lsof', '-i', '4tcp@0.0.0.0:37373'], executable = 'lsof'
preexec_fn = None, close_fds = False, cwd = None, env = None
universal_newlines = False, startupinfo = None, creationflags = 0, shell = False
to_close = set([19]), p2cread = None, p2cwrite = None, c2pread = 19
c2pwrite = 20, errread = None, errwrite = None

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       cwd, env, universal_newlines,
                       startupinfo, creationflags, shell, to_close,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite):
        """Execute program (POSIX version)"""

        if isinstance(args, types.StringTypes):
            args = [args]
        else:
            args = list(args)

        if shell:
            args = ["/bin/sh", "-c"] + args
            if executable:
                args[0] = executable

        if executable is None:
            executable = args[0]

        def _close_in_parent(fd):
            os.close(fd)
            to_close.remove(fd)

        # For transferring possible exec failure from child to parent
        # The first char specifies the exception type: 0 means
        # OSError, 1 means some other error.
        errpipe_read, errpipe_write = self.pipe_cloexec()
        try:
            try:
                gc_was_enabled = gc.isenabled()
                # Disable gc to avoid bug where gc -> file_dealloc ->
                # write to stderr -> hang.  http://bugs.python.org/issue1336
                gc.disable()
                try:
                    self.pid = os.fork()
                except:
                    if gc_was_enabled:
                        gc.enable()
                    raise
                self._child_created = True
                if self.pid == 0:
                    # Child
                    try:
                        # Close parent's pipe ends
                        if p2cwrite is not None:
                            os.close(p2cwrite)
                        if c2pread is not None:
                            os.close(c2pread)
                        if errread is not None:
                            os.close(errread)
                        os.close(errpipe_read)

                        # When duping fds, if there arises a situation
                        # where one of the fds is either 0, 1 or 2, it
                        # is possible that it is overwritten (#12607).
                        if c2pwrite == 0:
                            c2pwrite = os.dup(c2pwrite)
                        if errwrite == 0 or errwrite == 1:
                            errwrite = os.dup(errwrite)

                        # Dup fds for child
                        def _dup2(a, b):
                            # dup2() removes the CLOEXEC flag but
                            # we must do it ourselves if dup2()
                            # would be a no-op (issue #10806).
                            if a == b:
                                self._set_cloexec_flag(a, False)
                            elif a is not None:
                                os.dup2(a, b)
                        _dup2(p2cread, 0)
                        _dup2(c2pwrite, 1)
                        _dup2(errwrite, 2)

                        # Close pipe fds.  Make sure we don't close the
                        # same fd more than once, or standard fds.
                        closed = { None }
                        for fd in [p2cread, c2pwrite, errwrite]:
                            if fd not in closed and fd > 2:
                                os.close(fd)
                                closed.add(fd)

                        if cwd is not None:
                            os.chdir(cwd)

                        if preexec_fn:
                            preexec_fn()

                        # Close all other fds, if asked for - after
                        # preexec_fn(), which may open FDs.
                        if close_fds:
                            self._close_fds(but=errpipe_write)

                        if env is None:
                            os.execvp(executable, args)
                        else:
                            os.execvpe(executable, args, env)

                    except:
                        exc_type, exc_value, tb = sys.exc_info()
                        # Save the traceback and attach it to the exception object
                        exc_lines = traceback.format_exception(exc_type,
                                                               exc_value,
                                                               tb)
                        exc_value.child_traceback = ''.join(exc_lines)
                        os.write(errpipe_write, pickle.dumps(exc_value))

                    # This exitcode won't be reported to applications, so it
                    # really doesn't matter what we return.
                    os._exit(255)

                # Parent
                if gc_was_enabled:
                    gc.enable()
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)

            # Wait for exec to fail or succeed; possibly raising exception
            data = _eintr_retry_call(os.read, errpipe_read, 1048576)
            pickle_bits = []
            while data:
                pickle_bits.append(data)
                data = _eintr_retry_call(os.read, errpipe_read, 1048576)
            data = "".join(pickle_bits)
        finally:
            if p2cread is not None and p2cwrite is not None:
                _close_in_parent(p2cread)
            if c2pwrite is not None and c2pread is not None:
                _close_in_parent(c2pwrite)
            if errwrite is not None and errread is not None:
                _close_in_parent(errwrite)

            # be sure the FD is closed no matter what
            os.close(errpipe_read)

        if data != "":
            try:
                _eintr_retry_call(os.waitpid, self.pid, 0)
            except OSError as e:
                if e.errno != errno.ECHILD:
                    raise
            child_exception = pickle.loads(data)
>           raise child_exception
E           OSError: [Errno 2] No such file or directory

/nix/store/2nsq3cpqy514n5raqdzpfh6lprpqlmjk-python-2.7.13/lib/python2.7/subprocess.py:1024: OSError
===================== 2 failed, 519 passed in 1.78 seconds =====================

[jluttine]

Ok, I had to install lsof, that fixed the second failure. The first failure is still open.

[jluttine]

Installing geoip Python package didn't help.

[jluttine]

Oh, actually I installed some geoip system package, not the Python package. Installing the Python package probably solves this. I'll try that and then close this issue if it works.

[jluttine]

Yep, installing GeoIP Python package fixed the issue.

[meejah]

Did you install the "dev" extra? I think that would have covered GeoIP. That is pip install --editable .[dev]

[meejah]

Could I change the documentation somewhere to make this more-obvious?

[jluttine]

From my point of view, the confusion came from that it didn't raise an import error but just told a weird FailTest: 'NoneType' != 'GeoIP' error. I understand that GeoIP is optional and in the code it's tried to import but if not found, then set to None. Perhaps, this test could do import GeoIP before anything else so that the test would raise an import error, which is much more informative?

From documentation point of view, dev-requirements.txt is a good place to have this information, and it's there. From the error, I didn't first realize that I could be just missing some dependency.