Descriptors leak router

[alejandro-colomar]

@andrey-zelenkov

Hi Andrei!

I've installed python3-pytest in Debian Sid, and run pytest on the master branch, and I get the famous report.

$ pytest
================================== test session starts ==================================
platform linux -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/alx/src/nginx/unit/master
collected 863 items / 2 skipped                                                         

test/test_access_log.py ssssssssssssssssss                                        [  2%]
test/test_asgi_application.py ssssssssssssssssssssss                              [  4%]
test/test_asgi_application_unix_abstract.py s                                     [  4%]
test/test_asgi_lifespan.py sssss                                                  [  5%]
test/test_asgi_targets.py sssss                                                   [  5%]
test/test_asgi_websockets.py sssssssssssssssssssssssssssssssssssss                [ 10%]
test/test_client_ip.py ssssssss                                                   [ 11%]
test/test_configuration.py ssssssssssssssssssssssssssssssss                       [ 14%]
test/test_forwarded_header.py ssssssss                                            [ 15%]
test/test_go_application.py ssssssssss                                            [ 16%]
test/test_go_isolation.py sssssssss                                               [ 17%]
test/test_go_isolation_rootfs.py s                                                [ 18%]
test/test_http_header.py ssssssssssssssssssssssssssssssss                         [ 21%]
test/test_java_application.py sssssssssssssssssssssssssssssssssssssss             [ 26%]
test/test_java_isolation_rootfs.py s                                              [ 26%]
test/test_java_websockets.py ssssssssssssssssssssssssssssssss                     [ 30%]
test/test_njs.py sssss                                                            [ 30%]
test/test_njs_modules.py ssss                                                     [ 31%]
test/test_node_application.py ssssssssssssssssssssssssssssss                      [ 34%]
test/test_node_es_modules.py sss                                                  [ 34%]
test/test_node_websockets.py sssssssssssssssssssssssssssssssss                    [ 38%]
test/test_perl_application.py ssssssssssssssssssssssssss                          [ 41%]
test/test_php_application.py ssssssssssssssssssssssssssssssssssssssssssssssss     [ 47%]
test/test_php_basic.py sssssss                                                    [ 48%]
test/test_php_isolation.py ss                                                     [ 48%]
test/test_php_targets.py ss                                                       [ 48%]
test/test_proxy.py sssssssssssssssss                                              [ 50%]
test/test_proxy_chunked.py sssss                                                  [ 51%]
test/test_python_application.py sssssssssssssssssssssssssssssssssssssssss         [ 55%]
test/test_python_basic.py ssssssss                                                [ 56%]
test/test_python_environment.py sssssss                                           [ 57%]
test/test_python_isolation.py ssssss                                              [ 58%]
test/test_python_isolation_chroot.py s                                            [ 58%]
test/test_python_procman.py sssssssssssss                                         [ 60%]
test/test_python_targets.py ss                                                    [ 60%]
test/test_reconfigure.py .E.                                                      [ 60%]
test/test_reconfigure_tls.py ssss                                                 [ 60%]
test/test_respawn.py sss                                                          [ 61%]
test/test_response_headers.py ..s....                                             [ 62%]
test/test_return.py .....                                                         [ 62%]
test/test_rewrite.py ....s...                                                     [ 63%]
test/test_routing.py ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 70%]
sssssssssssssssssssssssssssssssssssssssssssssssssssss                             [ 76%]
test/test_routing_tls.py s                                                        [ 76%]
test/test_ruby_application.py sssssssssssssssssssssssssssssssssssssss             [ 81%]
test/test_ruby_hooks.py ssssss                                                    [ 82%]
test/test_ruby_isolation.py s                                                     [ 82%]
test/test_settings.py ssssssssssssssss                                            [ 84%]
test/test_static.py ................s                                             [ 85%]
test/test_static_chroot.py ...........                                            [ 87%]
test/test_static_fallback.py .....s.                                              [ 88%]
test/test_static_mount.py sss                                                     [ 88%]
test/test_static_share.py ...                                                     [ 88%]
test/test_static_symlink.py ...                                                   [ 89%]
test/test_static_types.py ........                                                [ 90%]
test/test_static_variables.py ......                                              [ 90%]
test/test_status.py sssss                                                         [ 91%]
test/test_status_tls.py s                                                         [ 91%]
test/test_tls.py ssssssssssssssssssssssss                                         [ 94%]
test/test_tls_conf_command.py ss                                                  [ 94%]
test/test_tls_sni.py sssssssss                                                    [ 95%]
test/test_unix_abstract.py ss                                                     [ 95%]
test/test_upstreams_rr.py sssssssssssss                                           [ 97%]
test/test_usr1.py ss                                                              [ 97%]
test/test_variables.py ....................s.                                     [100%]

======================================== ERRORS =========================================
_________________________ ERROR at teardown of test_reconfigure _________________________

request = <SubRequest 'run' for <Function test_reconfigure>>

    @pytest.fixture(autouse=True)
    def run(request):
        unit = unit_run()

        option.skip_alerts = [
            r'read signalfd\(4\) failed',
            r'sendmsg.+failed',
            r'recvmsg.+failed',
        ]
        option.skip_sanitizer = False

        _fds_info['main']['skip'] = False
        _fds_info['router']['skip'] = False
        _fds_info['controller']['skip'] = False

        yield

        # stop unit

        error_stop_unit = unit_stop()
        error_stop_processes = stop_processes()

        # prepare log

        with Log.open() as f:
            log = f.read()
            Log.set_pos(f.tell())

        if not option.save_log and option.restart:
            shutil.rmtree(unit['temp_dir'])
            Log.set_pos(0)

        # clean temp_dir before the next test

        if not option.restart:
            _clear_conf(log=log)
            _clear_temp_dir()

        # check descriptors

>       _check_fds(log=log)

test/conftest.py:242: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test/unit/log.py:17: in inner_function
    raise exception
test/unit/log.py:14: in inner_function
    func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    @print_log_on_assert
    def _check_fds(*, log=None):
        def waitforfds(diff):
            for _ in range(600):
                fds_diff = diff()

                if fds_diff <= option.fds_threshold:
                    break

                time.sleep(0.1)

            return fds_diff

        ps = _fds_info['main']
        if not ps['skip']:
            fds_diff = waitforfds(
                lambda: _count_fds(unit_instance['pid']) - ps['fds']
            )
            ps['fds'] += fds_diff

            assert fds_diff <= option.fds_threshold, 'descriptors leak main process'

        else:
            ps['fds'] = _count_fds(unit_instance['pid'])

        for name in ['controller', 'router']:
            ps = _fds_info[name]
            ps_pid = ps['pid']
            ps['pid'] = pid_by_name(ps['name'])

            if not ps['skip']:
                fds_diff = waitforfds(lambda: _count_fds(ps['pid']) - ps['fds'])
                ps['fds'] += fds_diff

                if not option.restart:
                    assert ps['pid'] == ps_pid, f'same pid {name}'

>               assert fds_diff <= option.fds_threshold, f'descriptors leak {name}'
E               AssertionError: descriptors leak router
E               assert 67 <= 0
E                +  where 0 = <unit.option.Options object at 0x7f11bcccb710>.fds_threshold

test/conftest.py:541: AssertionError
------------------------------- Captured stdout teardown --------------------------------
Path to unit.log:
/tmp/unit-test-g9pcxams/unit.log

================================ short test summary info ================================
ERROR test/test_reconfigure.py::test_reconfigure - AssertionError: descriptors leak router
================== 94 passed, 771 skipped, 1 error in 71.82s (0:01:11) ==================

It should be easy to reproduce in a Debian Sid machine.

[andrey-zelenkov]

Hi @alejandro-colomar,

Sorry, but Debian Sid is not in the list of our official packages.

[alejandro-colomar]

Sure, I know. Just saying it because there are other (non-GNU/Linux) platforms that you support and reproduce this bug (or at least did some time ago; e.g., FreeBSD), and now it's reproducible in a GNU/Linux system. I expect that soon other distributions will start reproducing it (maybe Fedora or Ubuntu?).

[andrey-zelenkov]

I just checked our FreeBSD, Fedora, and Ubuntu machines, and this test passed successfully. Overall, we have a great packaging team, and they integrate fresh OS images once they become available in our testing environment. So, if this issue is easy to trigger, I believe we should be able to catch it. Thank you for your care!

[alejandro-colomar]

Good to know. Thanks!