raw readinto() returned invalid length 4494852078 (should have been between 0 and 2297167872)

[ThomasWaldmann]

Bug report

A python file read crashes weirdly with raw readinto() returned invalid length 4494852078 (should have been between 0 and 2297167872).

See there:

https://github.com/borgbackup/borg/issues/6140#issuecomment-1120387922

That "returned invalid length" is even more than the total file length.

It is currently rather unclear what the root cause of this is, this is why I am asking for help / clarification here.

Considering it crashes at a sanity check within the python stdlib, I suspect the bug is either somewhere deeper in CPython or in some lower layer (like fs or os / kernel / hardware level).

Your environment

• CPython versions tested on: 3.7
• Operating system and architecture: Linux 64bit / ZFS (Hetzner Storage Box, Server hw with ECC memory)

Note:

I of course already tried to find more information about this kind of error doing a web search. But all I found are cases of:

• network reads (but in our cases we have a normal fs read and the fs is locally attached to the storage box as Hetzner told us)
• terminal / text encoding / crlf related stuff, also does not match our case

I guess this is the place that raises this exception:

• https://github.com/python/cpython/blob/v3.7.1/Modules/_io/bufferedio.c#L1493
• https://github.com/python/cpython/blob/v3.10.4/Modules/_io/bufferedio.c#L1499

[serhiy-storchaka]

Here is the minimal code which performs the same sequence of reading operations:

import io
path = 'gh93287.dat'
with open(path, 'wb') as f:
    f.truncate(18+2297167872)

with open(path, 'rb') as f:
    data = f.read(18)
    assert len(data) == 18
    length = f.seek(0, io.SEEK_END)
    assert length == 18+2297167872
    f.seek(18, io.SEEK_SET)
    data = f.read(2297167872)
    assert len(data) == 2297167872

It passes on my computer. Does it fail on your computer? Does it fail if comment out writing operations and set path to the path of your file that causes error?

[ThomasWaldmann]

I am not the user experiencing this issue, I just brought it here because we could not solve it yet. The issue happens on a so called "storage box" of hosting company hetzner, there is no shell access, so as a user we can not run custom code.

They have borg backup installed on the storage box and borg users use this in client/server mode to make their backups to the storage box.

[ThomasWaldmann]

Also, there are quite many borg users using this borg release on a lot of different machines and usually do not run into this issue, so this is somehow special. But what exactly makes this fail here is still mysterious.

[ThomasWaldmann]

About the "pending" label: I already said we can't run own code on hetzner storage box.

But guess that does not hold any python dev back reviewing the code whether there is some size-related bug in Python. It does not seem to be a hardware / specific machine related bug, they have ECC memory and they also moved to another machine and the bug is still happening.

[debuglevel]

Not sure if this helps anyone, but just noting in case it could:

as of today (2024-08-25) this still happens on "Hetzner StorageBox":

• FreeBSD 14.0, OpenZFS 2.2.0
• Python 3.9.18
• Borg 1.2.7
• it happens only on a "subaccount", not the "main account".