arendst / Tasmota

Alternative firmware for ESP8266 and ESP32 based devices with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Full documentation at
https://tasmota.github.io/docs
GNU General Public License v3.0
21.69k stars 4.72k forks source link

Berry: fix file.readbytes() for large files #21657

Open Staars opened 1 week ago

Staars commented 1 week ago

Description:

Reading binary files into bytes in Berry leads to data corruption for large files (I did not figure out the threshold) by creating a bytes buffer of the correct size, that has correct data at the front but only holds zeros at the tail end. This fails silently and does not throw any error.

This seems to be related to the resize operation after the actual data read from the file system.

To reproduce we need a test file on the ESP, that is not too small. I used this one: https://github.com/Staars/MockUp/blob/main/watermeter.jpg

Then load in the Berry web console:

f = open("watermeter.jpg","r")
b =f.readbytes()
f.close()

Check the terminating bytes, which should be "FFD9" for a JPG file with:

b[-2..]

The current version shows "0000". Tested on ESP32 and ESP32-S3.

Unrelated to this PR the C6 crashes with heap corruption: CORRUPT HEAP: Bad head at 0x408407e8. Expected 0xabba1234 got 0x52844590

Checklist:

NOTE: The code change must pass CI tests. Your PR cannot be merged unless tests pass

sfromis commented 1 week ago

Looks like some much wider issue with non-small files. I tried to transfer a moderately sized file (ftp.be) with both Manage File system upload (web or via curl), or the UrlFetch command, on a board with PSRAM available, and this failed. Closest to a symptom was "file too large". I did manage to transfer a somewhat smaller .be file, 11 KB instead of failing with 17 KB.

s-hadinger commented 1 week ago

I'm afraid there is a deeper bug in the bytes.resize function. Let's put on hold until I see more clearly what is happening