Meaningful-Data / vtlengine

A Validation and Transformation Language engine, written in Python
https://docs.vtlengine.meaningfuldata.eu/
GNU Affero General Public License v3.0
7 stars 0 forks source link

Check Comparison operators execution time #47

Open mla2001 opened 1 week ago

mla2001 commented 1 week ago

Every time a comparison operator goes out in a serial scalar operation, it takes so long that it is because the element-by-element operation is redundant, launching the elements one by one and scaling as many times as there are elements in the series. Operations are now only applied once to the entire series and when scaling.

Hierarchical clauses were too slow, rather than using combine in the application of the hierarchical function. Now the combine is only applied based on a mask.

mla2001 commented 1 week ago

Evidence Before: image

After: image

This shows that it now only takes about 67% of the average time it did before optimization.