search

tensorflow / models

Models and examples built with TensorFlow
Other
76.98k stars 45.78k forks source link

eval.py RuntimeWarning: invalid value encountered in true_divide num_images_correctly_detected_per_class / num_gt_imgs_per_class) #5461

Closed Shashi42 closed 5 years ago

Shashi42 commented 5 years ago

System information

Describe the problem

I'm new to this object detection stuff I could run object_detection/eval.py for recognizing digits (0-9) but when i try the same for recognizing english alphabets i get this error.

WARNING:root:image 0 does not have groundtruth difficult flag specified
WARNING:root:The following classes have no ground truth examples: 6
[2 2 2 2 2 0 4 2 2 4 4 2 2 2 4 2 2 2 4 2 3 1 2 1 1 1] [2. 2. 0. 0. 2. 0. 4. 0. 2. 4. 0. 2. 2. 2. 2. 0. 2. 2. 4. 2. 2. 1. 1. 1.
 0. 1.]
C:\Users\shash\Desktop\alphabets\Anirudh\models_master\research\object_detection\utils\metrics.py:143: RuntimeWarning: invalid value encountered in true_divide
  num_images_correctly_detected_per_class / num_gt_imgs_per_class)

I made sure to change all the paths and number of classes in pipeline config and all the labels are correct in labelmap.pbtxt and in generate_tf_record.py

Source code / logs

metrics.py

I was trying to find the problem so i included some print statements. This function is where i got the issue

def compute_cor_loc(num_gt_imgs_per_class,
                    num_images_correctly_detected_per_class):
  """Compute CorLoc according to the definition in the following paper.

  https://www.robots.ox.ac.uk/~vgg/rg/papers/deselaers-eccv10.pdf

  Returns nans if there are no ground truth images for a class.

  Args:
    num_gt_imgs_per_class: 1D array, representing number of images containing
        at least one object instance of a particular class
    num_images_correctly_detected_per_class: 1D array, representing number of
        images that are correctly detected at least one object instance of a
        particular class

  Returns:
    corloc_per_class: A float numpy array represents the corloc score of each
      class
  """
  print(num_gt_imgs_per_class,num_images_correctly_detected_per_class)
  return np.where(
      num_gt_imgs_per_class == 0, np.nan,
      num_images_correctly_detected_per_class / num_gt_imgs_per_class)

My data:

label_map.pbtxt

item{
    name:"A"
    id:1
}
item{
    name:"B"
    id:2
}
item{
    name:"C"
    id:3
}
item{
    name:"D"
    id:4
}
item{
    name:"E"
    id:5
}
item{
    name:"F"
    id:6
}
item{
    name:"G"
    id:7
}
item{
    name:"H"
    id:8
}
item{
    name:"I"
    id:9
}
item{
    name:"J"
    id:10
}
item{
    name:"K"
    id:11
}
item{
    name:"L"
    id:12
}
item{
    name:"M"
    id:13
}
item{
    name:"N"
    id:14
}
item{
    name:"O"
    id:15
}
item{
    name:"P"
    id:16
}
item{
    name:"Q"
    id:17
}
item{
    name:"R"
    id:18
}
item{
    name:"S"
    id:19
}
item{
    name:"T"
    id:20
}
item{
    name:"U"
    id:21
}
item{
    name:"V"
    id:22
}
item{
    name:"W"
    id:23
}
item{
    name:"X"
    id:24
}
item{
    name:"Y"
    id:25
}
item{
    name:"Z"
    id:26
}

generate_tf_record.py

from __future__ import division
from __future__ import print_function
from __future__ import absolute_import

import os
import io
import pandas as pd
import tensorflow as tf
import sys

sys.path.append('C:/Users/shash/Desktop/alphabets/Anirudh/models_master/research/')
sys.path.append('C:/Users/shash/Desktop/alphabets/Anirudh/models_master/research/slim')

from PIL import Image
from object_detection.utils import dataset_util
from collections import namedtuple, OrderedDict

flags = tf.app.flags
flags.DEFINE_string('csv_input', 'C:/Users/shash/Desktop/alphabets/data/Eval.csv', 'Path to the CSV input')
flags.DEFINE_string('output_path', 'C:/Users/shash/Desktop/alphabets/data/Eval.record', 'Path to output TFRecord')
FLAGS = flags.FLAGS

# TO-DO replace this with label map
def class_text_to_int(row_label):
    #if row_label == 'Building':
    #    return 1
    #elif row_label == 'ParkingLot':
    #    return 2
    #elif row_label == 'Trees':
    #   return 3
    #------
    #if row_label == 'face':
    #    return 1
    #else:
    #    None
    if row_label == 'A':
        return 1
    elif row_label == 'B':
        return 2
    elif row_label == 'C':
        return 3
    elif row_label == 'D':
        return 4
    elif row_label == 'E':
        return 5
    elif row_label == 'F':
        return 6
    elif row_label == 'G':
        return 7
    elif row_label == 'H':
        return 8
    elif row_label == 'I':
        return 9
    elif row_label == 'J':
        return 10
    elif row_label == 'K':
        return 11
    elif row_label == 'L':
        return 12
    elif row_label == 'M':
        return 13
    elif row_label == 'N':
        return 14
    elif row_label == 'O':
        return 15
    elif row_label == 'P':
        return 16
    elif row_label == 'Q':
        return 17
    elif row_label == 'R':
        return 18
    elif row_label == 'S':
        return 19
    elif row_label == 'T':
        return 20
    elif row_label == 'U':
        return 21
    elif row_label == 'V':
        return 22
    elif row_label == 'W':
        return 23
    elif row_label == 'X':
        return 24
    elif row_label == 'Y':
        return 25
    elif row_label == 'Z':
        return 26
    else:
        None

def split(df, group):
    data = namedtuple('data', ['filename', 'object'])
    gb = df.groupby(group)
    return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path):
    with tf.gfile.GFile(os.path.join(path, '{}.jpg'.format(group.filename)), 'rb') as fid:
        encoded_jpg = fid.read()
    encoded_jpg_io = io.BytesIO(encoded_jpg)
    image = Image.open(encoded_jpg_io)
    width, height = image.size

    filename = str(group.filename).encode('utf-8','ignore')
    image_format = b'jpg'
    xmins = []
    xmaxs = []
    ymins = []
    ymaxs = []
    classes_text = []
    classes = []

    for index, row in group.object.iterrows():
        xmins.append(row['xmin'] / width)
        xmaxs.append(row['xmax'] / width)
        ymins.append(row['ymin'] / height)
        ymaxs.append(row['ymax'] / height)
        classes_text.append(row['class'].encode('utf8'))
        classes.append(class_text_to_int(row['class']))

    tf_example = tf.train.Example(features=tf.train.Features(feature={
        'image/height': dataset_util.int64_feature(height),
        'image/width': dataset_util.int64_feature(width),
        'image/filename': dataset_util.bytes_feature(filename),
        'image/source_id': dataset_util.bytes_feature(filename),
        'image/encoded': dataset_util.bytes_feature(encoded_jpg),
        'image/format': dataset_util.bytes_feature(image_format),
        'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
        'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
        'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
        'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
        'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
        'image/object/class/label': dataset_util.int64_list_feature(classes),
    }))
    return tf_example

def main(_):
    writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
    path = 'C:\\Users\\shash\\Desktop\\alphabets\\data\\images\\Eval'
    examples = pd.read_csv(FLAGS.csv_input)
    grouped = split(examples, 'filename')
    for group in grouped:
        tf_example = create_tf_example(group, path)
        writer.write(tf_example.SerializeToString())

    writer.close()
    output_path = FLAGS.output_path
    print('Successfully created the TFRecords: {}'.format(output_path))

if __name__ == '__main__':
    tf.app.run()
# -*- coding: utf-8 -*-
"""
Created on Tue Mar  6 16:48:21 2018

@author: nrc
"""

"""
Usage:
  # From tensorflow/models/
  # Create train data:
  python generate_tfrecord.py --csv_input=data/train_labels.csv  --output_path=train.record
  # Create test data:
  python generate_tfrecord.py --csv_input=data/test_labels.csv  --output_path=test.record
"""

pipeline.config

# Faster R-CNN with Inception v2, configuration for MSCOCO Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.

model {
  faster_rcnn {
    num_classes: 26
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 500
        max_dimension: 1024
      }
    }
    feature_extractor {
      type: 'faster_rcnn_inception_v2'
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 300
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}

train_config: {
  batch_size: 1
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.0002
          schedule {
            step: 900000
            learning_rate: .00002
          }
          schedule {
            step: 1200000
            learning_rate: .000002
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "C:\\Users\\shash\\Desktop\\alphabets\\Anirudh\\faster_rcnn_inception_v2_coco_2018_01_28\\faster_rcnn_inception_v2_coco_2018_01_28\\model.ckpt"
  from_detection_checkpoint: true
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the COCO dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps: 16000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "C:\\Users\\shash\\Desktop\\alphabets\\data\\Train.record"
  }
  label_map_path: "C:\\Users\\shash\\Desktop\\alphabets\\data\\label_map.pbtxt"
}

eval_config: {
  num_examples: 30
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 1
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "C:\\Users\\shash\\Desktop\\alphabets\\data\\Eval.record"
  }
  label_map_path: "C:\\Users\\shash\\Desktop\\alphabets\\data\\label_map.pbtxt"
  shuffle: false
  num_readers: 1
}

Training is going well (currently at 669 step) I'm kinda stuck here and worried if it might affect loss in the future . Any help or advice would be appreciated. Thank you.

tensorflowbutler commented 5 years ago

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. What is the top-level directory of the model you are using Bazel version Exact command to reproduce

Shashi42 commented 5 years ago

Sorry about that ,here's the rest of the information:

Shashi42 commented 5 years ago

So i was able to solve this issue.. Apparently true divide Run time Warning can occur when your eval data doesn't contain one or more specific examples. like in my case class 6 : F I forgot to add images of 'F' into eval folder. I did and now it's working. Closing the issue.

Manish-rai21bit commented 5 years ago

This is an issue. I do not think it is appropriate to if some class is missing from the eval set. it should default to some value rather than throwing a warning.