Failed pool allocation

[pleonard212]

We're seeing this on occasion -- have tried setting the threads to both the exact # of cores as well as a few less:

Traceback (most recent call last):
  File "segment_ydn_images.py", line 731, in <module>
    for result in pool_two.imap(segment_images, process_ids):
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 673, in next
    raise value
IndexError: cannot do a non-empty take from an empty axes.

[duhaime]

Ah, that's interesting. Let's sync up tomorrow to see where that empty array crops up. Multiprocessing pools hide most of the stacktrace, which makes it hard to identify the exact line that's causing an error, but we can wrap the full worker function (segment_images()) with:

try:
  function internals
except IndexError as exc:
  print(some_file_reference, exc)

so we can pinpoint the troublesome files and figure out why we're getting an empty numpy array...