search

openworm / behavioral_syntax

behavioral syntax analysis based on the paper of Andre Brown
Other
5 stars 1 forks source link

optimization #30

Closed ghost closed 7 years ago

ghost commented 8 years ago
  1. cythonize what can be cythonized. start here: http://docs.cython.org/src/userguide/numpy_tutorial.html#numpy-tutorial
  2. use itertools or some better alternative to loops in Python
  3. consider using PyParallel: http://www.infoq.com/articles/PyParallel

The following program for example: https://github.com/AidanRocke/behavioral_syntax/blob/master/models/discrete_models/discretization/posture_probabilities.py

takes about 2 mins to run for just one video with about 20000 frames. A heatmap of temporal posture distribution represents only a fraction of the analysis that will be done. Not even 5%.

Now, if we're considering a sample size of at least 30 different worms this would mean that the time taken to produce a lab report would be greater than 2 hours. And that's really the lower bound.

MichaelCurrie commented 8 years ago

In my opinion I think 2 hours is okay if we are comparing 30 different worms. OWAT still takes ~1 minute to process features for a single 4-minute worm video. The original Schafer Lab code was at least an order of magnitude worse.

Before worrying about optimizing I would want to know:

1) How long did André's Matlab code take to do the same analysis? 2) How long would the full analysis of all worm videos (i.e. 10 TB of videos) we are interested in take on a normal computer?

If the answer to 1 is "a similar amount of time" and 2 is "less than a week", I think we can deploy the virtual resources it would take to perform the analysis in a reasonable amount of time. AWS has very large instances we could use to bring the time down.

All I'm saying is, getting the thing done and integrated with OWAT to me should be accomplished before we worry about optimization.