kuanb / peartree

peartree: A library for converting transit data into a directed graph for sketch network analysis.
MIT License
201 stars 23 forks source link

[performance] Utilize mp.BaseManager to share reference object among processes #60

Closed kuanb closed 6 years ago

kuanb commented 6 years ago

Fixes https://github.com/kuanb/peartree/issues/59

Work towards dealing with https://github.com/kuanb/peartree/issues/12

Utilizes the BaseManager class to enable the multiprocessing starmap operation's mapped function to access a shared object containing read-only reference data needed by the function being run in parallel.

kapture 2018-04-10 at 21 44 57 Short GIF capturing htop output while peartree is run on a sample GTFS feed in a docker container with access to 2 CPUs.

Performance of run_etl.py profiler script (runs graph extraction on AC Transit feed) on 2016 MBP:

Parallelized, using 4 CPUs: 39s
Serial (no parallelization): 50s

H/T to @bryanculbertson for pointing me in the right direction with this.

kuanb commented 6 years ago

Note: Tests err'ing at tests/test_paths.py::test_synthetic_network make.

Error is related to writing to memory process does not own: make: *** [test] Abort trap: 6

codecov[bot] commented 6 years ago

Codecov Report

Merging #60 into master will decrease coverage by 1.13%. The diff coverage is 96.46%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
- Coverage    92.8%   91.66%   -1.14%     
==========================================
  Files          10       11       +1     
  Lines         639      672      +33     
==========================================
+ Hits          593      616      +23     
- Misses         46       56      +10
Impacted Files Coverage Δ
peartree/paths.py 95.65% <ø> (ø) :arrow_up:
peartree/toolkit.py 95.93% <88.88%> (+0.13%) :arrow_up:
peartree/summarizer.py 90% <90%> (-7.08%) :arrow_down:
peartree/parallel.py 97.87% <97.87%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 226fe71...1939147. Read the comment docs.