search

biigle / maia

:m: BIIGLE module for the Machine Learning Assisted Image Annotation method
GNU General Public License v3.0
2 stars 3 forks source link

Gracefully handle truncated images #30

Closed mzur closed 5 years ago

mzur commented 5 years ago

Skip truncated images during novelty detection and instance segmentation. Errors below.

Novelty detection (job ID 7):

Error while executing python command '/var/www/vendor/biigle/maia/src/config/../resources/scripts/novelty-detection/DetectionRunner.py /var/www/storage/maia_jobs/maia-7-novelty-detection/input.json':
Traceback (most recent call last):
  File "/var/www/vendor/biigle/maia/src/config/../resources/scripts/novelty-detection/DetectionRunner.py", line 120, in <module>
    runner.run()
  File "/var/www/vendor/biigle/maia/src/config/../resources/scripts/novelty-detection/DetectionRunner.py", line 57, in run
    clusters = images.make_clusters(number=self.clusters)
  File "/var/www/vendor/biigle/maia/src/resources/scripts/novelty-detection/ImageCollection.py", line 53, in make_clusters
    pca_features = np.array(list(pca_features))
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 556, in result_iterator
    yield future.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/www/vendor/biigle/maia/src/resources/scripts/novelty-detection/ImageCollection.py", line 10, in extract_image_pca_features
    return image.extract_pca_features()
  File "/var/www/vendor/biigle/maia/src/resources/scripts/novelty-detection/Image.py", line 37, in extract_pca_features
    return np.array(self._get_resized_image()).flatten()
  File "/var/www/vendor/biigle/maia/src/resources/scripts/novelty-detection/Image.py", line 34, in _get_resized_image
    return img.resize((width, height), PIL.Image.BILINEAR)
  File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 1782, in resize
    self.load()
  File "/usr/local/lib/python3.5/dist-packages/PIL/ImageFile.py", line 236, in load
    len(b))
OSError: image file is truncated (92 bytes not processed)

Instance segmentation (job ID 8):

Error while executing python command '/var/www/vendor/biigle/maia/src/config/../resources/scripts/instance-segmentation/InferenceRunner.py /var/www/storage/maia_jobs/maia-8-instance-segmentation/input-inference.json /var/www/storage/maia_jobs/maia-8-instance-segmentation/output-dataset.json /var/www/storage/maia_jobs/maia-8-instance-segmentation/output-training.json':
Using TensorFlow backend.

Configurations:
AUGMENTATION                   Sometimes(p=Binomial(Deterministic(float 0.50000000)), name=UnnamedSometimes, then_list=Sequential(name=UnnamedSometimes-then, random_order=False, children=[Fliplr(name=UnnamedFliplr, parameters=[Binomial(Deterministic(float 0.50000000))], deterministic=False), Flipud(name=UnnamedFlipud, parameters=[Binomial(Deterministic(float 0.50000000))], deterministic=False), Affine(name=UnnamedAffine, parameters=[Deterministic(float 1.00000000), Deterministic(int 0), Uniform(Deterministic(int -180), Deterministic(int 180)), Deterministic(float 0.00000000), Deterministic(int 1), Deterministic(int 0), Deterministic(constant), auto], deterministic=False), GaussianBlur(name=UnnamedGaussianBlur, parameters=[Uniform(Deterministic(float 0.00000000), Deterministic(float 2.00000000))], deterministic=False)], deterministic=False), else_list=Sequential(name=UnnamedSometimes-else, random_order=False, children=[], deterministic=False), deterministic=False)
BACKBONE                       resnet101
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE         None
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.7
DETECTION_NMS_THRESHOLD        0.3
FPN_CLASSIF_FC_LAYERS_SIZE     1024
GPU_COUNT                      1
GRADIENT_CLIP_NORM             5.0
IMAGES_PER_GPU                 1
IMAGE_CHANNEL_COUNT            3
IMAGE_MAX_DIM                  1024
IMAGE_META_SIZE                14
IMAGE_MIN_DIM                  64
IMAGE_MIN_SCALE                0
IMAGE_RESIZE_MODE              pad64
IMAGE_SHAPE                    [1024 1024    3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'mrcnn_class_loss': 1.0, 'mrcnn_mask_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'rpn_bbox_loss': 1.0, 'rpn_class_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [147.95957908 164.82751785 173.19284938]
MINI_MASK_SHAPE                (56, 56)
NAME                           maia_inference
NUM_CLASSES                    2
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
PRE_NMS_LIMIT                  6000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (32, 64, 128, 256, 512)
RPN_ANCHOR_STRIDE              1
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TOP_DOWN_PYRAMID_SIZE          256
TRAIN_BN                       False
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               0
WEIGHT_DECAY                   0.0001

WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
2019-01-11 23:16:50.811200: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-01-11 23:16:51.368945: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-11 23:16:51.369295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:07.0
totalMemory: 15.90GiB freeMemory: 15.61GiB
2019-01-11 23:16:51.369355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-01-11 23:16:51.666487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-01-11 23:16:51.666556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-01-11 23:16:51.666565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-01-11 23:16:51.666810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15123 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:07.0, compute capability: 6.0)
Image 1 of 54 (#171024)
Image 2 of 54 (#171031)
Image 3 of 54 (#171023)
Image 4 of 54 (#171027)
Image 5 of 54 (#171035)
Image 6 of 54 (#171034)
Image 7 of 54 (#171022)
Image 8 of 54 (#171038)
Image 9 of 54 (#171039)
Image 10 of 54 (#171037)
Image 11 of 54 (#171041)
Image 12 of 54 (#171042)
Image 13 of 54 (#171043)
Image 14 of 54 (#171044)
Image 15 of 54 (#171045)
Image 16 of 54 (#171046)
Image 17 of 54 (#171047)
Image 18 of 54 (#171028)
Image 19 of 54 (#171049)
Image 20 of 54 (#171050)
Image 21 of 54 (#171051)
Image 22 of 54 (#171052)
Image 23 of 54 (#171053)
Image 24 of 54 (#171025)
Image 25 of 54 (#171055)
Image 26 of 54 (#171056)
Image 27 of 54 (#171054)
Image 28 of 54 (#171058)
Image 29 of 54 (#171059)
Image 30 of 54 (#171060)
Image 31 of 54 (#171061)
Image 32 of 54 (#171062)
Image 33 of 54 (#171026)
Image 34 of 54 (#171064)
Image 35 of 54 (#171065)
Image 36 of 54 (#171066)
Image 37 of 54 (#171067)
Image 38 of 54 (#171068)
Image 39 of 54 (#171069)
Image 40 of 54 (#171070)
Image 41 of 54 (#171071)
Image 42 of 54 (#171072)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/skimage/io/_plugins/pil_plugin.py", line 53, in pil_to_ndarray
    image.getdata()[0]
  File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 1253, in getdata
    self.load()
  File "/usr/local/lib/python3.5/dist-packages/PIL/ImageFile.py", line 236, in load
    len(b))
OSError: image file is truncated (92 bytes not processed)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/www/vendor/biigle/maia/src/config/../resources/scripts/instance-segmentation/InferenceRunner.py", line 108, in <module>
    runner.run()
  File "/var/www/vendor/biigle/maia/src/config/../resources/scripts/instance-segmentation/InferenceRunner.py", line 65, in run
    image, info =  job.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/www/vendor/biigle/maia/src/config/../resources/scripts/instance-segmentation/InferenceRunner.py", line 76, in load_image
    return self.dataset.load_image(i), self.dataset.image_info[i]
  File "/var/www/vendor/biigle/maia/src/resources/scripts/instance-segmentation/mrcnn/utils.py", line 358, in load_image
    image = skimage.io.imread(self.image_info[image_id]['path'])
  File "/usr/local/lib/python3.5/dist-packages/skimage/io/_io.py", line 62, in imread
    img = call_plugin('imread', fname, plugin=plugin, **plugin_args)
  File "/usr/local/lib/python3.5/dist-packages/skimage/io/manage_plugins.py", line 214, in call_plugin
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/skimage/io/_plugins/pil_plugin.py", line 37, in imread
    return pil_to_ndarray(im, dtype=dtype, img_num=img_num)
  File "/usr/local/lib/python3.5/dist-packages/skimage/io/_plugins/pil_plugin.py", line 61, in pil_to_ndarray
    raise ValueError(error_message)
ValueError: Could not load "" 
Reason: "image file is truncated (92 bytes not processed)"
Please see documentation at: http://pillow.readthedocs.org/en/latest/installation.html#external-libraries
mzur commented 5 years ago

The previous fix didn't work. It still crashes when TensorFlow attempts to load the image.

mzur commented 5 years ago

The new attempt seems to work.