search

kuanb / peartree

peartree: A library for converting transit data into a directed graph for sketch network analysis.
MIT License
201 stars 23 forks source link

[performance] Leverage MP when interpolating trip stop times #80

Closed kuanb closed 6 years ago

kuanb commented 6 years ago

Fixes https://github.com/kuanb/peartree/issues/79

kuanb commented 6 years ago

Something's up:

Original run time was 92.79 seconds Now, with MP on: 465.67 seconds And with MP off: 254.18 seconds

So, somehow went to 4 min serialized and 7+ min parallelized. Need to look into what is causing this.

codecov[bot] commented 6 years ago

Codecov Report

Merging #80 into master will increase coverage by 0.45%. The diff coverage is 96.1%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #80      +/-   ##
=========================================
+ Coverage   90.85%   91.3%   +0.45%     
=========================================
  Files          12      13       +1     
  Lines         776     828      +52     
=========================================
+ Hits          705     756      +51     
- Misses         71      72       +1
Impacted Files Coverage Δ
peartree/parallel/route_processor.py 97.87% <100%> (ø)
peartree/parallel/trip_times_interpolator.py 95.34% <95.34%> (ø)
peartree/summarizer.py 91.21% <96.87%> (+1.28%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3778d4e...6d72ca0. Read the comment docs.

kuanb commented 6 years ago

Hrm. So it looks like perf is more like:

w/ AC Transit No MP: 146.8s Yes MP: 214.3s

And meanwhile old version still runs in 92.8s. Not good.

kuanb commented 6 years ago

Removal of df appending gets us down to 80s and 100s for no MP and MP. Which is acceptable since AC is a smaller reference layer. Gonna be happy with this for now.