mindflayer / python-mocket

a socket mock framework - for all kinds of socket animals, web-clients included
BSD 3-Clause "New" or "Revised" License
279 stars 41 forks source link

"Too many open files" error when making many mocked async requests #217

Closed ento closed 6 months ago

ento commented 6 months ago

Reproducible example that I think is minimal:

import httpx
from mocket.mockhttp import Entry
from mocket import async_mocketize

@pytest.mark.asyncio
@async_mocketize
async def test_many():
    url = "http://httpbin.local/ip"
    Entry.single_register(Entry.GET, url, status=404)

    for i in range(200):
        async with httpx.AsyncClient() as client:
            response = await client.get(url)
            assert response.status_code == 404

With number of open files limited in order to make it easier to hit the limit:

ulimit -n 100

This fails like so :

.devenv/state/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py:1627: in connect_tcp
    await get_running_loop().create_connection(
/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/asyncio/base_events.py:1085: in create_connection
    raise exceptions[0]
/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/asyncio/base_events.py:1069: in create_connection
    sock = await self._connect_sock(
/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/asyncio/base_events.py:973: in _connect_sock
    await self.sock_connect(sock, address)
/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/asyncio/selector_events.py:632: in sock_connect
    self._sock_connect(fut, sock, address)
/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/asyncio/selector_events.py:640: in _sock_connect
    fd = sock.fileno()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    @staticmethod
    def fileno():
>       Mocket.r_fd, Mocket.w_fd = os.pipe()
E       OSError: [Errno 24] Too many open files

mocket/mocket.py:260: OSError
mindflayer commented 6 months ago

Here is the new version which fixes this issue: https://pypi.org/project/mocket/3.12.3/