Tinche / aiofiles

File support for asyncio
Apache License 2.0
2.66k stars 150 forks source link

`write` result is inaccurate when multi-coroutine is executed #135

Closed xiaoxinmiao closed 2 years ago

xiaoxinmiao commented 2 years ago

Inaccurate results on windows when mode=a is set and Semaphore is 2

Environment:

system: windows aiofiles version: 0.8.0

Test

test1:

sem = asyncio.Semaphore(2) When the parameter is passed in 1, the result is correct When the parameter is passed in 2, the result is incorrect

test2

Wrong result on windows, correct result on linux

Source

import asyncio
import os

import aiofiles

async def write_chunk_to_file(sem: asyncio.Semaphore, chunk: str) -> None:
    async with sem:
        async with aiofiles.open("test.txt", mode="a") as f:
            await f.write(f'{chunk}\n')

async def main() -> None:
    # Prepare the data to write in batches
    data_chunks: list[str] = [
        "Berlin", "Germany", "Madrid", "Spain", "Rome", "Italy",
        "Washington D.C.", "USA", "Ottawa", "Canada", "Mexico City", "Mexico",
        "Tokyo", "Japan", "Beijing", "China", "Manila", "Philippines"
    ]

    # Truncate existing file
    file_path = 'test.txt'
    if os.path.isfile(file_path):
        os.remove(file_path)

    # Launch coroutines to write batches to the file
    sem = asyncio.Semaphore(1)
    await asyncio.gather(*[write_chunk_to_file(sem, chunk) for chunk in data_chunks])

if __name__ == "__main__":
    asyncio.run(main())

Result

The result of running on windows when setting Semaphore to 2

expect data


1. Berlin
2. Germany
3. Madrid
4. Spain
5. Rome
6. Italy
7. Washington D.C.
8. USA
9. Ottawa
10. Canada
11. Mexico City
12. Mexico
13. Tokyo
14. Japan
15. Beijing
16. China
17. Manila
18. Philippines

actual data

1. Berlin
2. Germany
3. Madrid
4. Italy
5. Washington D.C.
6. Ottawa
7. Mexico City
8. Tokyo
9. Beijing
10. China
11. Philippines

动画2

Tinche commented 2 years ago

You're probably running into a race condition caused by the combination of Python and the operating system, I don't think I can fix this in aiofiles. I would suggest using a lock (or a semaphore with a value of 1) to mitigate the issue.

xiaoxinmiao commented 2 years ago

@Tinche ok,by the way, is it reliable to run on linux and macOs