Unexpected Error while using my own dataset.

[MikeXlYang]

I'm using MICCAI 2017 EndoVis to do instance segmentation on surgical tools. The datasets includes 1800 images(25 of which have none instances and were excluded). I write my own tool.py as shown below. But when I started training the follow error occurs and the training process doesn't stop until sever iterations. The same error happened while another user using 8-channel images. But I'm only using simple 3-channel ones. I could really use some help as I'm pretty bad in tf and python. Really appreciate your help. ERROR:root:Error processing image {'id': '7frame032', 'path': '/home/user/MaskRCNN/Mask_RCNN-master/ReArranged_origin/Left_frames/7frame032.png', 'source': 'tools'} Traceback (most recent call last): File "/home/user/.local/lib/python3.5/site-packages/mask_rcnn-2.1-py3.5.egg/mrcnn/model.py", line 1704, in data_generator use_mini_mask=config.USE_MINI_MASK) File "/home/user/.local/lib/python3.5/site-packages/mask_rcnn-2.1-py3.5.egg/mrcnn/model.py", line 1227, in load_image_gt mask = utils.resize_mask(mask, scale, padding, crop) File "/home/user/.local/lib/python3.5/site-packages/mask_rcnn-2.1-py3.5.egg/mrcnn/utils.py", line 517, in resize_mask mask = scipy.ndimage.zoom(mask, zoom=[scale, scale, 1], order=0) File "/home/user/.local/lib/python3.5/site-packages/scipy/ndimage/interpolation.py", line 573, in zoom zoom = _ni_support._normalize_sequence(zoom, input.ndim) File "/home/user/.local/lib/python3.5/site-packages/scipy/ndimage/_ni_support.py", line 65, in _normalize_sequence

And the following are my tools.py `import os import sys import json import datetime import numpy as np import skimage.io from imgaug import augmenters as iaa

Root directory of the project

ROOT_DIR = os.path.abspath("../../")

Import Mask RCNN

sys.path.append(ROOT_DIR) # To find local version of the library from mrcnn.config import Config from mrcnn import utils from mrcnn import model as modellib from mrcnn import visualize

Path to trained weights file

COCO_WEIGHTS_PATH = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")

Directory to save logs and model checkpoints, if not provided

through the command line argument --logs

DEFAULT_LOGS_DIR = os.path.join(ROOT_DIR, "logs")

Results directory

Save submission files here

RESULTS_DIR = os.path.join(ROOT_DIR, "results/tools/")

Under Linux

RESULTS_DIR = os.path.join(ROOT_DIR, r"results\tools")

To acquire val sets, I choose to segment the sequences with

constant gaps.

templenum = 0 templename = '' templename2 = '' valname = '' fullidname = '' VAL_IMAGE_IDS = [] for idcount in range(1,226): if idcount % 9 == 1: templenum = str(idcount) while len(templenum) < 3: templenum = '0' + templenum VAL_IMAGE_IDS.append('frame' + str(templenum))

WRONG_IMAGE_IDS = ['5frame092','5frame093','7frame077', '7frame078','7frame079','7frame081', '7frame086','7frame092','7frame093', '7frame094','7frame095','7frame137', '7frame138','7frame139','7frame140', '7frame141','7frame144','7frame157', '7frame158','7frame159'] ############################################################

Configurations

############################################################

class ToolsConfig(Config): """Configuration for training on the nucleus segmentation dataset."""

Give the configuration a recognizable name

NAME = "tools"

# Adjust depending on your GPU memory
IMAGES_PER_GPU = 1

# Number of classes (including background)
NUM_CLASSES = 1 + 6  # Background + nucleus

USE_MINI_MASK = False
# Backbone network architecture
# Supported values are: resnet50, resnet101
BACKBONE = "resnet101"

# Input image resizing
# Random crops of size 512x512

class ToolsDataset(utils.Dataset):

def load_tools(self, dataset_dir, subset):
    """Load a subset of the nuclei dataset.
    dataset_dir: Root directory of the dataset
    subset: Subset to load. Either train or val
    """
    # Add classes. We have one class.
    # Naming the dataset nucleus, and the class nucleus
    self.add_class("tools", 1, "Bipolar_Forceps")
    self.add_class("tools", 2, "Curved_Scissors")
    self.add_class("tools", 3, "Grasping_Retractor")
    self.add_class("tools", 4, "Needle_Driver")
    self.add_class("tools", 5, "Prograsp_Forceps")
    self.add_class("tools", 6, "Vessel_Sealer")

    # Which subset?
    # "val": use hard-coded list above
    # "train": use data from stage1_train minus the hard-coded list above
    # else: use the data from the specified sub-directory
    assert subset in ["train", "val"]
    trimage_ids = []
    # validation set usage
    if subset == "val":            
        for imgcount in range(1,9):
            for valname in VAL_IMAGE_IDS:
                if valname not in WRONG_IMAGE_IDS:
                    templename = str(imgcount) + valname
                    trimage_ids.append(templename)
        for trimage_id in trimage_ids:
            self.add_image(
                    "tools",
                    image_id = trimage_id,
                    path = os.path.join(dataset_dir,"Left_frames", trimage_id + ".png"))
    # training set usage
    elif subset == 'train':
        # Get image ids from directory names
        for imgcount in range(1,9):
            for templenum in range(225):
                templename2 = str(templenum)
                while len(templename2) < 3:
                    templename2 = '0' + templename2
                fullidname = 'frame' + templename2
                if fullidname not in WRONG_IMAGE_IDS:
                    fullidname = str(imgcount) + fullidname
                    if fullidname not in WRONG_IMAGE_IDS:
                        trimage_ids.append(fullidname)
        for trimage_id in trimage_ids:
            self.add_image(
                    "tools",
                    image_id = trimage_id,
                    path = os.path.join(dataset_dir,"Left_frames", trimage_id + ".png"))

def load_mask(self, image_id):
    """Generate instance masks for an image.
   Returns:
    masks: A bool array of shape [height, width, instance count] with
        one mask per instance.
    class_ids: a 1D array of class IDs of the instance masks.
    """
    # os.listdir(dataset_dir)
    instance_names = ['Prograsp_Forceps_Right', 'Grasping_Retractor_Right',
                      'Bipolar_Forceps', 'Prograsp_Forceps_Left',
                      'Needle_Driver_Right', 'Vessel_Sealer_Right',
                      'Bipolar_Forceps_Left', 'Needle_Driver_Left', 
                      'Prograsp_Forceps', 
                      'Grasping_Retractor_Left', 'Curved_Scissors', 
                      'Grasping_Retractor', 'Vessel_Sealer']
    instancedic = {'Prograsp_Forceps_Right':5 , 'Grasping_Retractor_Right':3,
                      'Bipolar_Forceps':1, 'Prograsp_Forceps_Left':5,
                      'Needle_Driver_Right':4, 'Vessel_Sealer_Right':6,
                      'Bipolar_Forceps_Left':1, 'Needle_Driver_Left':4, 
                      'Prograsp_Forceps':5, 
                      'Grasping_Retractor_Left':3, 'Curved_Scissors':2, 
                      'Grasping_Retractor':3, 'Vessel_Sealer':6}

    mask_dirs = []
    mask = []
    classids = []
    info = self.image_info[image_id]
    for instance_name in instance_names:
        mask_dirs = os.path.join(ROOT_DIR,"ReArranged_origin",
                                     instance_name, info['id'] + ".png" )
        m = skimage.io.imread(mask_dirs).astype(np.bool)
        if np.sum(m) > 0:
            mask.append(m)
            classids.append(instancedic[instance_name])                
    if np.sum(mask) > 0:            
        mask = np.stack(mask, axis=-1)
        classids = np.stack(classids)
    elif np.sum(mask) == 0:
        print(info['id'])
    # Return mask, and array of class IDs of each instance. Since we have
    # one class ID, we return an array of ones
    return mask, classids

def image_reference(self, image_id):
    """Return the path of the image."""
    info = self.image_info[image_id]
    if info["source"] == "tools":
        return info["id"]
    else:
        super(self.__class__, self).image_reference(image_id)

############################################################

Training

############################################################

def train(model, dataset_dir, subset): """Train the model."""

Training dataset.

dataset_train = ToolsDataset()
dataset_train.load_tools(dataset_dir, "train")
dataset_train.prepare()

# Validation dataset
dataset_val = ToolsDataset()
dataset_val.load_tools(dataset_dir, "val")
dataset_val.prepare()

# If starting from imagenet, train heads only for a bit
# since they have random weights
print("Train network heads")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=20,
            layers='heads')

print("Train all layers")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=40,
            layers='all')

config = ToolsConfig() config.display() model = modellib.MaskRCNN(mode="training", config=config, model_dir=DEFAULT_LOGS_DIR) weights_path = COCO_WEIGHTS_PATH model.load_weights(weights_path, by_name=True, exclude=[ "mrcnn_class_logits", "mrcnn_bbox_fc", "mrcnn_bbox", "mrcnn_mask"]) datasetsdir = os.path.join(ROOT_DIR,'ReArranged_origin') train(model, datasetsdir, 'train')`

[MikeXlYang]

Problem has been solved. Some of the images have no instances

[windson87]

Hey, I face the same issue. but I wonder why it takes only images with instances? Does it really make sense? For example, if I wanna train a binary class task, some images just have background, some have instances. For those simply contains background, it should also learn that type of instance-free images, isn't it? in my tasks, instances are very less than images with simply background,

[thomasyue]

@MikeXlYang I'm facing the same problem. How did you deal with images with no instances? Do you delete them directly?

[MikeXlYang]

Yes I simply added an if judgement before loading an image.

---Original--- From: "thomasyue"notifications@github.com Date: Wed, May 8, 2019 23:13 PM To: "matterport/Mask_RCNN"Mask_RCNN@noreply.github.com; Cc: "Mike Xilin Yang"1113670776@qq.com;"Mention"mention@noreply.github.com; Subject: Re: [matterport/Mask_RCNN] Unexpected Error while using my own dataset. (#829)

@MikeXlYang I'm facing the same problem. How did you deal with images with no instances? Do you delete them directly?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[thomasyue]

@MikeXlYang So if the mask is empty, you won't train the image, correct? In this case, if the model is predicting a test image without instance, will it able to detect?