(migrated from Jason's NLeSC/PattyGenerate/issue#28)
The current KeyMatchFull does a single threaded all-to-all compare of all image keysets. To parallalise this I created a KeyMatchPart which does a one-to-all compare. This KeyMatchPart can then easily be run in parallel using a python thread pool.
Unfortunately, this approach increases the memory usage by a factor N (where N is the number of processes) since each process loads its own copy of the keys. Also, the key loading time is increased by a factor X/2 (where X is the number of images) since each process loads (part of) the keys over and over again.
For large datasets this will become a problem. We already see this occuring on a (video based) dataset of approx. 5K images.
To solve this we need a mutlithreaded version of KeyMatchFull that loads the data once, and then schedules all compare jobs internally using N threads.
jmaassen
commented
9 years ago
(migrated from Jason's NLeSC/PattyGenerate/issue#28)
The current KeyMatchFull does a single threaded all-to-all compare of all image keysets. To parallalise this I created a KeyMatchPart which does a one-to-all compare. This KeyMatchPart can then easily be run in parallel using a python thread pool.
Unfortunately, this approach increases the memory usage by a factor N (where N is the number of processes) since each process loads its own copy of the keys. Also, the key loading time is increased by a factor X/2 (where X is the number of images) since each process loads (part of) the keys over and over again.
For large datasets this will become a problem. We already see this occuring on a (video based) dataset of approx. 5K images.
To solve this we need a mutlithreaded version of KeyMatchFull that loads the data once, and then schedules all compare jobs internally using N threads.