search

tobozo / ESP32-targz

🗜️ An Arduino library to unpack/uncompress tar, gz, and tar.gz files on ESP32 and ESP8266
Other
117 stars 15 forks source link

Crash trying to untar #64

Closed Anyeos closed 1 year ago

Anyeos commented 1 year ago

When I use an intermediate file it gunzip successfully but when I try to untar anything it crash: crash_log.txt

Anyway I don't understand why it only shows two rows because on the tar there are more files. The directory structure of the tar is as follows:

css/
css/bootstrap.min.css
css/bootstrap-theme.min.css
index.htm
js/
js/jquery-3.6.1.min.js
js/bootstrap.min.js

It only extracts the only one file that is displayed on the log. The file is extracted corrupted. Original file: bootstrap.min.css.txt Extracted file: bootstrap.min.css.txt

tobozo commented 1 year ago

hi, thanks for your feedback :+1:

Can you please provide an archive file and code snippet so I can reproduce the issue?

I'll also need more contextual information:

tobozo commented 1 year ago

I performed a few tests and observations with tarGzStreamExpander() on a Wemos Mini D1, also found a sweet spot:

Now the crash log you provided indicates the ESP was already low on heap when the expanding occured, the expander selected NO dictionary, but the dictionary is mandatory for gz->tar->filesystem streaming: decompressing without a dictionary requires a seekable gz source file, but streams don't allow seeking.

I'll work on a fix so that a proper error is emitted (e.g. Not enough heap, use an intermediate file) instead of starting a decompression with an invalid context.

Anyeos commented 1 year ago

Hello, I just was starting to understanding the ESP8266 (I know other MCUs) and now I am more sure that it may be related with not sufficient memory because I observed some other behaviours like that.

But I don't choose "No dictionary" that was automatic so I guessed it was correct because the developer of that library knows what is doing. Anyway I tested with dictionary and same behaviour.

I excuse me because I don't have the time to give you more details. That was on a complex code (ESPASyncWeb and so) that becomes too much to show here as an code that raise the error. I simply followed the provided examples I don't do nothing special with ESP32-targz and when it failed I simply let it out.

Debug had not been enabled so that was not the problem and I just had a /heap GET defined to just read the free available heap over time, and that showed around 40K always. So I guessed it was not the heap but who knows because it is async maybe the WebService were occupying too much memory?

Now I have a more stable working code and can test newly. In some days may I will test it again. I will test lowering GZIP_BUFF_SIZE then, because as I said before I didn't touch anything I just followed the examples.

As a filesystem I don't use SPIFFS because it is buggy for everything so I discarded it. Instead I use LittleFS and it is working as I said because I have an already working device with webpages and it actually is working flawlessly.

I hope your library works because it is a great library to upload one large file in one shot.

Aah, the versions that I am using are the latest ones that the Arduino IDE can download.

Thank you for your answer and may I will test it another day. See you.

tobozo commented 1 year ago

well thanks to you I found a couple of bugs and also a way to reduce s bit more the memory footprint on esp8266.

I also found out tarGzStreamExpander() can work when the esp8266 has just booted, but glitches as soon as the free heap goes under 4Kb.

Latest changes are visible on this branch.

Other possible strategies:

tobozo commented 1 year ago

memory improvements for esp8266 in latest release should fix this, feel free to reopen if necessary