search

heavyai / rbc

Remote Backend Compiler
https://rbc.readthedocs.io
BSD 3-Clause "New" or "Revised" License
30 stars 10 forks source link

Reduce compile-time #560

Closed guilhermeleobas closed 1 year ago

guilhermeleobas commented 1 year ago

get_called_functions and DTypeComparison.run_pass where responsible for 80% of the compile-time in a single test:

Before:

``` _ ._ __/__ _ _ _ _ _/_ Recorded: 00:46:45 Samples: 50578 /_//_/// /_\ / //_// / //_'/ // Duration: 54.667 CPU time: 55.333 / _/ v4.4.0 Program: pytest -rs -sv --tb=line -x rbc/tests/heavydb/test_nrt_string.py -k capitalize 54.165 pytest_pyfunc_call _pytest/python.py:187 └─ 54.165 test_string_methods rbc/tests/heavydb/test_nrt_string.py:164 └─ 54.165 RemoteHeavyDB.sql_execute rbc/heavydb/remoteheavydb.py:852 └─ 53.804 RemoteHeavyDB.register rbc/heavydb/remoteheavydb.py:1269 └─ 53.799 RemoteHeavyDB._register rbc/heavydb/remoteheavydb.py:1274 └─ 53.122 compile_to_LLVM rbc/irtools.py:418 ├─ 32.919 get_called_functions rbc/irtools.py:52 │ └─ 32.795 get_called_functions rbc/irtools.py:52 │ ├─ 21.668 get_called_functions rbc/irtools.py:52 │ │ ├─ 10.613 get_called_functions rbc/irtools.py:52 │ │ │ ├─ 4.372 JITRemoteCodeLibrary.get_defined_functions numba/core/codegen.py:807 │ │ │ │ [192 frames hidden] numba, llvmlite, │ │ │ ├─ 3.608 get_called_functions rbc/irtools.py:52 │ │ │ │ ├─ 1.370 JITRemoteCodeLibrary.get_defined_functions numba/core/codegen.py:807 │ │ │ │ │ [167 frames hidden] numba, llvmlite, │ │ │ │ └─ 0.905 get_called_functions rbc/irtools.py:52 │ │ │ ├─ 0.838 ValueRef.opcode llvmlite/binding/value.py:295 │ │ │ │ [106 frames hidden] llvmlite, numba, , ctypes │ │ │ ├─ 0.680 ValueRef.name llvmlite/binding/value.py:143 │ │ │ │ [46 frames hidden] llvmlite, numba, │ │ │ └─ 0.601 _InstructionsIterator.__next__ llvmlite/binding/value.py:316 │ │ │ [121 frames hidden] llvmlite, numba, │ │ ├─ 8.003 JITRemoteCodeLibrary.get_defined_functions numba/core/codegen.py:807 │ │ │ [196 frames hidden] numba, llvmlite, │ │ ├─ 1.230 ValueRef.name llvmlite/binding/value.py:143 │ │ │ [46 frames hidden] llvmlite, numba, │ │ └─ 0.695 ValueRef.opcode llvmlite/binding/value.py:295 │ │ [106 frames hidden] llvmlite, numba, , ctypes │ ├─ 8.755 JITRemoteCodeLibrary.get_defined_functions numba/core/codegen.py:807 │ │ [196 frames hidden] numba, llvmlite, │ └─ 1.277 ValueRef.name llvmlite/binding/value.py:143 │ [46 frames hidden] llvmlite, numba, ├─ 19.399 compile_instance rbc/irtools.py:315 │ ├─ 14.731 compile_extra numba/core/compiler.py:690 │ │ [5968 frames hidden] numba, contextlib, llvmlite, └─ 0.765 JITRemoteCodeLibrary._optimize_final_module numba/core/codegen.py:667 [4 frames hidden] numba, llvmlite ```

After:

``` _ ._ __/__ _ _ _ _ _/_ Recorded: 00:44:48 Samples: 18110 /_//_/// /_\ / //_// / //_'/ // Duration: 21.513 CPU time: 22.342 / _/ v4.4.0 Program: pytest -rs -sv --tb=line -x rbc/tests/heavydb/test_nrt_string.py -k capitalize 21.005 pytest_pyfunc_call _pytest/python.py:187 └─ 21.005 test_string_methods rbc/tests/heavydb/test_nrt_string.py:164 └─ 21.005 RemoteHeavyDB.sql_execute rbc/heavydb/remoteheavydb.py:852 ├─ 20.755 RemoteHeavyDB.register rbc/heavydb/remoteheavydb.py:1269 │ └─ 20.750 RemoteHeavyDB._register rbc/heavydb/remoteheavydb.py:1274 │ ├─ 20.123 compile_to_LLVM rbc/irtools.py:424 │ │ ├─ 16.543 compile_instance rbc/irtools.py:320 │ │ │ ├─ 14.593 compile_extra numba/core/compiler.py:690 │ │ │ │ [36678 frames hidden] numba, contextlib, llvmlite, weakref,... │ │ │ └─ 1.844 get_called_functions rbc/irtools.py:57 │ │ │ ├─ 1.460 JITRemoteCodeLibrary.get_defined_functions numba/core/codegen.py:807 │ │ │ │ [176 frames hidden] numba, llvmlite, │ │ │ └─ 0.220 ValueRef.name llvmlite/binding/value.py:143 │ │ │ [46 frames hidden] llvmlite, numba, │ │ ├─ 3.147 get_called_functions rbc/irtools.py:57 │ │ │ └─ 3.030 get_called_functions rbc/irtools.py:57 │ │ │ ├─ 2.236 JITRemoteCodeLibrary.get_defined_functions numba/core/codegen.py:807 │ │ │ │ [187 frames hidden] numba, llvmlite, │ │ │ └─ 0.377 ValueRef.name llvmlite/binding/value.py:143 │ │ │ [46 frames hidden] llvmlite, numba, │ │ └─ 0.392 JITRemoteCodeLibrary._optimize_final_module numba/core/codegen.py:667 │ │ [4 frames hidden] numba, llvmlite │ ├─ 0.275 RemoteHeavyDB.thrift_call rbc/heavydb/remoteheavydb.py:549 │ │ └─ 0.275 Client.__call__ rbc/thrift/client.py:151 │ │ └─ 0.274 TClient._req thriftpy2/thrift.py:204 │ │ [5 frames hidden] thriftpy2, │ └─ 0.271 RemoteHeavyDB.targets rbc/remotejit.py:730 │ └─ 0.271 RemoteHeavyDB.retrieve_targets rbc/heavydb/remoteheavydb.py:1036 └─ 0.249 RemoteHeavyDB.thrift_call rbc/heavydb/remoteheavydb.py:549 └─ 0.249 Client.__call__ rbc/thrift/client.py:151 └─ 0.249 TClient._req thriftpy2/thrift.py:204 [5 frames hidden] thriftpy2, 0.352 import_module importlib/__init__.py:108 └─ 0.245 rbc/__init__.py:1 ```