search

P1sec / pycrate

A Python library to ease the development of encoders and decoders for various protocols and file formats; contains ASN.1 and CSN.1 compilers.
GNU Lesser General Public License v2.1
381 stars 132 forks source link

Nuitka compilation fails #122

Closed vazir closed 2 years ago

vazir commented 3 years ago

It's written in the main Readme Nuitka is supported, attempt to compile the example led to several hours of compilation (actually optimization of the TCAP clasees, and subsequent OOM kill of compilation process (after it has eaten ~30G of RAM :)

the main eater were: Nuitka:INFO:Doing module local optimizations for 'pycrate_asn1dir.TCAP_CAP'. and Nuitka:INFO:Doing module local optimizations for 'pycrate_asn1dir.TCAP_MAPv2v3'. ... Nuitka:INFO: Total memory usage before generating C code: 27.18 GB (29186654208 bytes): Nuitka:INFO: Total memory usage before running scons: 27.94 GB (30000025600 bytes):

./c-nuitka.sh: line 3: 2279604 Killed pipenv run python3 -m nuitka --show-progress --show-scons --standalone --follow-imports test.py

p1-bmu commented 3 years ago

Yes, Nuikta can be quite resource-consuming for the compilation stage !

Personally, I would not use it to try to compile the entire pycrate library, but only to compile an application that uses a part of it. But maybe can you give some more precision on what you tried to compile with it ? (you say the example, but I don't know which one...). If I remind correctly, nuikta follows all references, what may trigger some very large amount of code (and memory) when following e.g. the dict that references all ASN.1 modules, see: https://github.com/P1sec/pycrate/blob/master/pycrate_asn1c/specdir.py

Try to get rid of the part of pycrate you don't need (mainly the ASN.1 compiler and all ASN.1 modules you don't need). If this still does not work, there may be an issue with some recursion constructs existing within TCAP-MAP and TCAP-CAMEL ASN.1 definitions: if this is the case, I don't have clear advice, unless to remove problematic objects from the ASN.1 definitions.

vazir commented 3 years ago

Thank you :) actually the test code was just a few lines, from_ber(unhexlify(tcap_cap_packet) to_ber() Attempt to compile under nuitka was to compare the performance. Will try doing as you have suggested.

p1-bmu commented 3 years ago

Nuitka binaries did not perform really faster than the CPython runtime when I tested it (2 / 3 years ago). In case you are looking for a performance and speed improvement, I would recommend pypy3.