pysal / segregation

Segregation Measurement, Inferential Statistics, and Decomposition Analysis
https://pysal.org/segregation/
BSD 3-Clause "New" or "Revised" License
112 stars 26 forks source link

overhaul inference framework #174

Closed knaaptime closed 3 years ago

knaaptime commented 3 years ago
knaaptime commented 3 years ago

tagging @darribas and @ljwolf in case there is a better way to do the parallelization. Currently the threading backend is much faster for simulate_evenness and simulate_systematic_randomization, but multiprocessing seems to work much better for simulate_reallocation. The loky backend always crashes for any of them?

knaaptime commented 3 years ago

cc: @martinfleis

knaaptime commented 3 years ago

scratch that... the loky backend is fastest for all approaches, i just needed to update joblib and friends

knaaptime commented 3 years ago

@sjsrey @renanxcortes ok, this should be all set. We get a nice speedup in the inference classes thanks to the parallelization, the only side effect is that setting a random seed is a bit more complicated, so i opted to turn the test precision down to two decimals

codecov-commenter commented 3 years ago

Codecov Report

Merging #174 (dcae6bb) into master (403bcf6) will decrease coverage by 2.28%. The diff coverage is 61.57%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #174      +/-   ##
==========================================
- Coverage   65.77%   63.49%   -2.29%     
==========================================
  Files         116      118       +2     
  Lines        4322     4421      +99     
==========================================
- Hits         2843     2807      -36     
- Misses       1479     1614     +135     
Impacted Files Coverage Δ
segregation/network/network.py 25.58% <0.00%> (-1.70%) :arrow_down:
segregation/tests/test_batch_compute.py 100.00% <ø> (ø)
segregation/tests/test_multiscalar_profile.py 100.00% <ø> (ø)
segregation/util/network.py 84.31% <0.00%> (ø)
segregation/inference/randomization.py 21.62% <21.62%> (ø)
segregation/singlegroup/spatial_dissim.py 87.17% <50.00%> (ø)
segregation/inference/comparative.py 61.64% <61.64%> (ø)
segregation/batch/batch_compute.py 90.36% <75.75%> (-9.64%) :arrow_down:
segregation/inference/inference_wrappers.py 72.80% <78.43%> (-13.51%) :arrow_down:
segregation/_base.py 82.47% <80.00%> (-0.69%) :arrow_down:
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 403bcf6...dcae6bb. Read the comment docs.

knaaptime commented 3 years ago

excellent, thanks James. I was pretty sure #177 only touched infrastructure, so figured it was easiest to continue focusing on this PR while touches more. I wasnt sure if there would be conflicts and since 177 is smaller it seemed easier to rebase. I'm open to suggestions if you think it would be easier some other way though

jGaboardi commented 3 years ago

excellent, thanks James. I was pretty sure #177 only touched infrastructure, so figured it was easiest to continue focusing on this PR while touches more. I wasnt sure if there would be conflicts and since 177 is smaller it seemed easier to rebase. I'm open to suggestions if you think it would be easier some other way though

Yep, probably better to do #177 after this PR merged.