search

dmazzella / ufastlz

Micropython wrapper for FastLZ, a lightning-fast lossless compression library
14 stars 0 forks source link

ufastlz on ESP32-WROOM #1

Closed 963io closed 2 years ago

963io commented 2 years ago

Hi there!

I'm developing something on ESP32. Used idf.py build to include the ufastlz module. Had to make two tweaks: 1) Added to CMakeLists.txt code: idf_component_register(SRCS "modufastlz.c" "FastLZ/fastlz.c" INCLUDE_DIRS "FastLZ") And 2) Made tweaks to modufastlz.c as follows: .. STATIC const mp_rom_map_elem_t mp_module_ufastlz_globals_table[] = { {MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_fastlz)}, // removed excessive '_' .. and MP_REGISTER_MODULE(MP_QSTR_fastlz, mp_module_ufastlz, 1); // initialized directly

Now it compiled and even importing. fastlz.compress('aaa') gives an error "at least 16 chars", but fastlz.compress("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbb bbbbbbbbb") dies with

`Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandle d.

Core 1 register dump: PC : 0x40097598 PS : 0x00060b30 A0 : 0x800f6275 A1 : 0x3f fceb80 A2 : 0x00000000 A3 : 0x00000054 A4 : 0x00000005 A5 : 0x3f fe5ed0 A6 : 0x00000062 A7 : 0x62626262 A8 : 0x80097590 A9 : 0x3f fceb70 A10 : 0x00000003 A11 : 0x00060b23 A12 : 0x00060b20 A13 : 0x3f fcedd0 A14 : 0x3ffc6d44 A15 : 0x00000000 SAR : 0x00000018 EXCCAUSE: 0x00 00001c EXCVADDR: 0x0000005c LBEG : 0x4000c349 LEND : 0x4000c36b LCOUNT : 0xff ffffff

Backtrace:0x40097595:0x3ffceb800x400f6272:0x3ffceba0 0x40104278:0x3ffcebc0 0x401 05359:0x3ffcebe0 0x400dc48a:0x3ffcec00 0x400dcfa8:0x3ffcec20 0x400d1f82:0x3ffcec 40 0x400da3f1:0x3ffceca0 0x400e0739:0x3ffcecd0 0x400e0869:0x3ffcecf0 0x40084aa5: 0x3ffced10 0x400da390:0x3ffcedb0 0x400e0739:0x3ffcee10 0x400e0762:0x3ffcee30 0x4 01108ae:0x3ffcee50 0x40110be0:0x3ffceee0 0x400f229c:0x3ffcef20

ELF file SHA256: 0000000000000000

Rebooting...

` Causing reboot. What am I missing here?

dmazzella commented 2 years ago

Hi, try to change vstr_init with vstr_init_len.

Now i can't try, give me some feedback.

Thanks

963io commented 2 years ago

Thanks for quick reply, @dmazzella! Should I change it everywhere?

dmazzella commented 2 years ago

https://github.com/dmazzella/ufastlz/blob/5f1ddfc966bfe835afc50cc07f95a1cc31720437/modufastlz.c#L70

https://github.com/dmazzella/ufastlz/blob/5f1ddfc966bfe835afc50cc07f95a1cc31720437/modufastlz.c#L102

963io commented 2 years ago

Nope, same issue `>>> fastlz.compress("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbb") Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.

Core 1 register dump: PC : 0x40097598 PS : 0x00060530 A0 : 0x800f6275 A1 : 0x3ffceb80
A2 : 0x00000000 A3 : 0x00000054 A4 : 0x00000005 A5 : 0x3ffe5c80
A6 : 0x00000062 A7 : 0x62626262 A8 : 0x80097590 A9 : 0x3ffceb70
A10 : 0x00000003 A11 : 0x00060523 A12 : 0x00060520 A13 : 0x3ffcedd0
A14 : 0x3ffc6d44 A15 : 0x00000000 SAR : 0x00000018 EXCCAUSE: 0x0000001c
EXCVADDR: 0x0000005c LBEG : 0x4000c349 LEND : 0x4000c36b LCOUNT : 0xffffffff

Backtrace:0x40097595:0x3ffceb800x400f6272:0x3ffceba0 0x40104278:0x3ffcebc0 0x40105359:0x3ffcebe0 0x400dc48a:0x3ffcec00 0x400dcfa8:0x3ffcec20 0x400d1f82:0x3ffcec40 0x400da3f1:0x3ffceca0 0x 400e0739:0x3ffcecd0 0x400e0869:0x3ffcecf0 0x40084aa5:0x3ffced10 0x400da390:0x3ffcedb0 0x400e0739:0x3ffcee10 0x400e0762:0x3ffcee30 0x401108ae:0x3ffcee50 0x40110be0:0x3ffceee0 0x400f229c:0x 3ffcef20

ELF file SHA256: 0000000000000000

Rebooting... `

And yes, I did make clean and reflashed it

963io commented 2 years ago

In case it can help, during compilation it shows `/root/firmware/modules/ufastlz/FastLZ/fastlz.c: In function 'fastlz_memmove': /root/firmware/modules/ufastlz/FastLZ/fastlz.c:91:17: warning: this statement may fall through [-Wimplicit-fallthrough=] dest++ = src++;


/root/firmware/modules/ufastlz/FastLZ/fastlz.c:92:7: note: here
       case 2:
       ^~~~
/root/firmware/modules/ufastlz/FastLZ/fastlz.c:93:17: warning: this statement may fall through [-Wimplicit-fallthrough=]
         *dest++ = *src++;
         ~~~~~~~~^~~~~~~~
/root/firmware/modules/ufastlz/FastLZ/fastlz.c:94:7: note: here
       case 1:
       ^~~~
`
dmazzella commented 2 years ago

I can't reproduce it, I have only stm32 boards and this is the output:

MicroPython v1.18-355-g9ab66b50c-dirty on 2022-04-15; PYBD-SF6W with STM32F767IIK
Type "help()" for more information.
>>> import _fastlz as fastlz
>>> fastlz.compress("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbb")
b'\x01aa\xe0\x1f\x01\x00b\xe0\x07\x00\x04bbbbb'
>>>
963io commented 2 years ago

Any advice for porting it to esp32? I'm sorry, that's somewhat new to me, so any help is greatly appreciated!

dmazzella commented 2 years ago

You need fastlz for a specific needs or you can try to switch to uminiz?

Try to define FASTLZ_USE_MEMMOVE=0 in your build

963io commented 2 years ago

I need compression.

Tried uminiz. It compiled "alright" `[1351/1372] Building C object esp-idf/m...rmware/modules/uminiz/miniz/miniz.c.obj /root/firmware/modules/uminiz/miniz/miniz.c: In function 'mz_compress2': /root/firmware/modules/uminiz/miniz/miniz.c:325:46: warning: comparison is always false due to limited range of data type [-Wtype-limits] if ((mz_uint64)(source_len | pDest_len) > 0xFFFFFFFFU) ^ /root/firmware/modules/uminiz/miniz/miniz.c: In function 'mz_uncompress2': /root/firmware/modules/uminiz/miniz/miniz.c:568:48: warning: comparison is always false due to limited range of data type [-Wtype-limits] if ((mz_uint64)(pSource_len | *pDest_len) > 0xFFFFFFFFU) ^ [1356/1372] Building C object esp-idf/m...re/modules/uminiz/miniz/miniz_zip.c.obj /root/firmware/modules/uminiz/miniz/miniz_zip.c:161:9: note: #pragma message: Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.

pragma message("Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.")

     ^~~~~~~

/root/firmware/modules/uminiz/miniz/miniz_zip.c: In function 'mz_zip_writer_add_mem_ex_v2': /root/firmware/modules/uminiz/miniz/miniz_zip.c:3226:34: warning: comparison is always false due to limited range of data type [-Wtype-limits] if (((mz_uint64)buf_size > 0xFFFFFFFF) || (uncomp_size > 0xFFFFFFFF)) ^ [1369/1372] Building C object CMakeFile...re/modules/uminiz/miniz/miniz_zip.c.obj /root/firmware/modules/uminiz/miniz/miniz_zip.c:161:9: note: #pragma message: Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.

pragma message("Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.")

     ^~~~~~~

, but... >>> _miniz.compress('test test test test test test test teset') Traceback (most recent call last): File "", line 1, in ValueError: compress: -4`

Tried FASTLZ_USE_MEMMOVE=0 with previous results (dies on me)

dmazzella commented 2 years ago

_miniz.compress('test test test test test test test teset') Traceback (most recent call last): File "", line 1, in ValueError: compress: -4

MZ_MEM_ERROR = -4

malloc used in esp32 port has problem? without hardware I can't test it, I'm sorry ...