Open fredgido opened 1 year ago
Quick fix:
from aiofile.utils import LineReader
# 1 megabyte chunks
LineReader.CHUNK_SIZE = 2 ** 20
For complete fix the some time or help is wanted.
Pretty sure this still does a read of the size chunk size for each line anyway so increasing only makes slower
Long story short
LineReader is very slow
Expected behavior
LineReader is as fast as normal line reading. Read Chunk size actually prevents extra reads.
Actual behavior
LineReader is slow, takes many ms per line. LineReader causes a read for each line.
Steps to reproduce
Additional info
Sync version to compare:
My temporary solution that only works for python approved new lines from the file
__iter__
, its only twice as slow as sync version: