search

djsutherland / py-sdm

Python implementation of nonparametric nearest-neighbor-based estimators for divergences between distributions.
http://cs.cmu.edu/~dsutherl/sdm/
BSD 3-Clause "New" or "Revised" License
48 stars 8 forks source link

use flann's multithreading instead of multiprocessing in get_divs #19

Closed djsutherland closed 11 years ago

djsutherland commented 11 years ago

FLANN has OpenMP parallelization across the query points. So we could just stack all the query points into a matrix and move along columns of the div matrix, searching all the points against each index one-by-one.

This has probably less memory overhead, doesn't depend on fork() inheritance (so could work on windows), and could simplify a lot of the logic to progressbar callbacks and so on. (Though progressbars across pools will have to stay around for SVM parameter tuning.)

djsutherland commented 11 years ago

Solved in the features-class branch, mostly in 23b6a88.