search

tiny-crab / CellSwiper

Image classification software for machine learning algorithm training.
0 stars 1 forks source link

Make batch importing multithreaded #132

Open EvanWiederspan opened 7 years ago

EvanWiederspan commented 7 years ago

We imported a batch today that had 7000 pictures. It worked great, but the entire server hangs while processing, and this made it unusable for several minutes while it worked. We need to have the image processing for batch imports happen in another thread

AetherUnbound commented 7 years ago

It seems that multithreading with Node is similar to multithreading on Python: it can't really be done in the most basic sense. Our best bet might be to run a "cluster": https://nodejs.org/api/cluster.html#cluster_cluster This would allow the one worker thread to go off and downsample the images while another worker handles other requests.