P1sec / pycrate

A Python library to ease the development of encoders and decoders for various protocols and file formats; contains ASN.1 and CSN.1 compilers.
GNU Lesser General Public License v2.1
381 stars 132 forks source link

How to reset a class to it's empty default state #121

Closed vazir closed 2 years ago

vazir commented 3 years ago

Hello! Is there any way to reset a class to empty state? When first loaded, it has empty structure like TCAP_MAPv2v3.TCAP_MAP_Messages.TCAP_MAP_Message() gives None and obviously after initializing with any from_ber or whatever at gets the content, so how to clean it? Is there any root object which can be set to None or method, cleaning up everything?

Will setting the TCAP_MAPv2v3.TCAP_MAP_Messages.TCAP_MAP_Message._val = None be enough to clean it or there are more sub-objects to clean?

p1-bmu commented 3 years ago

When encoding / decoding complex objects, you set values into internal components, not only into the object itself. So, setting None in the object's value is not a complete clean-up.

There are 2 solutions to do so: 1) You can deepcopy() the ASN.1 object you want to use for encoding / decoding: each time you do so, you have an empty and clean object. This can however be quite costly, especially for big object such as TCAP-MAP-Message. 2) You can call the reset_val() method of the object. This will clean-up all internal values and components. https://github.com/P1sec/pycrate/blob/da5afc091ebae21674908d90a7a9584eca4b0f69/pycrate_asn1rt/asnobj.py#L1105

vazir commented 3 years ago

Thank you, it's what i have been looking for! deepcopy, (softly speaking), is a heavy operation, so heavy so it's almost unusable for production (single copy of a clean TCAP_CAP.TCAP_CAP_Messages.TCAP_CAP_Message takes 0.1 of a second!). It's much more feasible to use one-pass init, generate and than cleanup :) Seems cleanup with reset_val() and re-initializing from the object() structure 10 times faster than deepcopying.

Deepcopy of the clean TCAP_CAP.TCAP_CAP_Messages.TCAP_CAP_Message
Total func Deepcopy of the clean TCAP_CAP.TCAP_CAP_Messages.TCAP_CAP_Message - deepcopy time taken for 1000 iterations is: 100.030975, and 0.100031 per iteration which is 10 iters per second.

Anyway, using from_ber without the prior reset_val is much faster. Seems reset_val is also a quite heavy operation. initilizing the IDP with prior reset_val is roughly 10 times slower than without it. So another question - is reset_val actually requred, if re-initilizing object by set_val or from ber? Or it will rewrite the structure anyway?

With reset_val:

Decode BER and2Pickle
Total func Decode BER and2Pickle - complex_dec_pickle time taken for 1000 iterations is: 8.671490, and 0.008671 per iteration which is 115 iters per second.
Unpickle and encode BER
Total func Unpickle and encode BER - complex_enc_unpickle time taken for 1000 iterations is: 8.870809, and 0.008871 per iteration which is 113 iters per second.

without reset_val

Decode BER and2Pickle
Total func Decode BER and2Pickle - complex_dec_pickle time taken for 1000 iterations is: 0.320916, and 0.000321 per iteration which is 3116 iters per second.
Unpickle and encode BER
Total func Unpickle and encode BER - complex_enc_unpickle time taken for 1000 iterations is: 0.486498, and 0.000486 per iteration which is 2056 iters per second.

All speed tests, without reset_val

Testing decoding from bytes
Total func Testing decoding from bytes - from_ber time taken for 1000 iterations is: 0.313813, and 0.000314 per iteration which is 3187 iters per second.
Testing appying from object
Total func Testing appying from object - set_val time taken for 1000 iterations is: 0.068076, and 0.000068 per iteration which is 14690 iters per second.
Testing encoding to bytes
Total func Testing encoding to bytes - to_ber time taken for 1000 iterations is: 0.385943, and 0.000386 per iteration which is 2591 iters per second.
Testing conversion to json
Total func Testing conversion to json - to_jer time taken for 1000 iterations is: 0.169017, and 0.000169 per iteration which is 5917 iters per second.
Testing conversion from json
Total func Testing conversion from json - from_jer time taken for 1000 iterations is: 0.129839, and 0.000130 per iteration which is 7702 iters per second.
Testing getting object
Total func Testing getting object - (NoName) time taken for 1000 iterations is: 0.000176, and 0.000000 per iteration which is 5667978 iters per second.
Testing pickling dump
Total func Testing pickling dump - dumps time taken for 1000 iterations is: 0.003350, and 0.000003 per iteration which is 298506 iters per second.
Decode BER and encode JSON
Total func Decode BER and encode JSON - complex_decode time taken for 1000 iterations is: 0.479076, and 0.000479 per iteration which is 2087 iters per second.
Decode JSON and encode BER
Total func Decode JSON and encode BER - complex_encode time taken for 1000 iterations is: 0.534296, and 0.000534 per iteration which is 1872 iters per second.
Decode BER and2Pickle
Total func Decode BER and2Pickle - complex_dec_pickle time taken for 1000 iterations is: 0.320916, and 0.000321 per iteration which is 3116 iters per second.
Unpickle and encode BER
Total func Unpickle and encode BER - complex_enc_unpickle time taken for 1000 iterations is: 0.486498, and 0.000486 per iteration which is 2056 iters per second.
Objects equality: True
p1-bmu commented 3 years ago

Basically, in a single thread environment, you don't need to clean-up any ASN.1 objects and can use them for several sequential encoding / decoding as such. In case you are willing to do multi-threading, then having a deepcopy of each object per thread seems a good idea to me.