[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
import io
import polars as pl
fixture = """\
first_name,last_name
josé,österlung
additional,content
"""
k = 500 # does not trigger with k=50
long_fixture = fixture * k
# imagine `fixture_bytes` is a large file opened in binary mode:
fixture_bytes = io.BytesIO(long_fixture.encode("latin1"))
# so we need to wrap it in TextIOWrapper to present TextIO to polars:
wrapper = io.TextIOWrapper(fixture_bytes, "latin1", "replace")
pl.read_csv(wrapper)
yields:
OSError: failed to write whole buffer
Log output
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/tim/projects/polars-textwrapper-repro/.direnv/python-3.12/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tim/projects/polars-textwrapper-repro/.direnv/python-3.12/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tim/projects/polars-textwrapper-repro/.direnv/python-3.12/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tim/projects/polars-textwrapper-repro/.direnv/python-3.12/lib/python3.12/site-packages/polars/io/csv/functions.py", line 418, in read_csv
df = _read_csv_impl(
^^^^^^^^^^^^^^^
File "/Users/tim/projects/polars-textwrapper-repro/.direnv/python-3.12/lib/python3.12/site-packages/polars/io/csv/functions.py", line 564, in _read_csv_impl
pydf = PyDataFrame.read_csv(
^^^^^^^^^^^^^^^^^^^^^
OSError: failed to write whole buffer
Issue description
Collecting the data into a StringIO before passing it to polars with e.g.
I stumbled upon the same problem when trying to implement polars import/export features for pyexasol (Exasol Database client library).
According to the docs, TextIOWrapper both IO[str] | IO[bytes] should work with read_csv, so this is probably a bug?
Checks
Reproducible example
yields:
Log output
Issue description
Collecting the data into a StringIO before passing it to polars with e.g.
works, though is no longer a streaming operation.
Expected behavior
Should not crash.
Installed versions