IntelPython / sdc

Numba extension for compiling Pandas data frames, Intel® Scalable Dataframe Compiler
https://intelpython.github.io/sdc-doc/
BSD 2-Clause "Simplified" License
645 stars 61 forks source link

Adding debug decorator to print Numba compile stats #925

Closed kozlov-alexey closed 3 years ago

kozlov-alexey commented 3 years ago

This adds a decorator to print Numba compilation stats for a decorated function and all nested functions (literally all compiled overloads). Filtering can be made using decorator arguments.

NOTE: this may show a bit lesser times than compile time calculated using common approach via first_run_exec_time - second_run_exec_time.

pep8speaks commented 3 years ago

Hello @kozlov-alexey! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:

Comment last updated at 2020-11-10 12:34:39 UTC
kozlov-alexey commented 3 years ago

Example output (if level=0 arg is used):

Function: csv_reader_py$4
    Pipeline: nopython
    Time: 0.7950319000000032

Function: sdc_pandas_dataframe_drop_impl$34
    Pipeline: nopython
    Time: 2.902112299999988
...
kozlov-alexey commented 3 years ago

@Hardcode84 Do you think we can merge this? Some builds are failed due to new pyarrow version, which needs investigation, but this PR is not changing main functionality only adds debug possibilities and is not related to these issues.

Hardcode84 commented 3 years ago

Is this critical for us at the moment? I would prefer it to have at least some tests (e.g. run some test in separate process and match output format with regex)

kozlov-alexey commented 3 years ago

run some test in separate process and match output format with regex

@Hardcode84 I added some tests as you have suggested (haven't done that with separate process though - it seems easier to do this in the same test process).