I noticed that converting PDB files to parquet files was notably slow, so I utilized Scalene, a Python profiling tool, to analyze the performance bottlenecks. The profiler's output, problematic part highlighted in the screenshot below, shows significant time consumption in two specific lines of code involving Pandas operations, accounting for 62% of the execution time (blue bars).
In this merge request I have replaced these two lines using a python dictionary which is much faster.
This change made the conversion three times faster and the results are still the same.
I noticed that converting PDB files to parquet files was notably slow, so I utilized Scalene, a Python profiling tool, to analyze the performance bottlenecks. The profiler's output, problematic part highlighted in the screenshot below, shows significant time consumption in two specific lines of code involving Pandas operations, accounting for 62% of the execution time (blue bars).
In this merge request I have replaced these two lines using a python dictionary which is much faster. This change made the conversion three times faster and the results are still the same.