File open attempted even when too many files are open

[GiovanH]

I have a program where I read the first few bytes from a large number of files.

    async with aiofiles.open(filepath, "rb") as fp:
        magic = await fp.read(4)

However, I find that on windows, running this gives OSError: [Errno 24] Too many open files: '{filepath}'

Shouldn't await fp.read() also wait for the appropriate OS resources?

[BenjiTheC]

I believe you are running the operations in a aysncio.gather or asyncio.wait. I solve the problem by introducing a semaphore to limit the concurrent execution of this specific piece of code

async def main():
    sem = asyncio.Semaphore(num_of_max_files_open)

    coro_queue = []
    for file_path in file_paths:  # Large number of file
        coro_queue.append(asyncio.create_task(read_file_magic(file_path, sem))

    asyncio.gather(*coro_queue)

async def read_file_magic(file_path, sem):
    async with sem:
        async with aiofiles.open(file_path) as f:
            magic = await f.read(4)