Open magedhelmy1 opened 4 years ago
same error
Traceback (most recent call last):
File "model_main_tf2.py", line 113, in
same here
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.408593744]] [[0.384375]]
[[{{node Assert/AssertGuard/else/_49/Assert/AssertGuard/Assert}}]]
[[MultiDeviceIteratorGetNextFromShard]]
[[RemoteCall]]
same here
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.408593744]] [[0.384375]] [[{{node Assert/AssertGuard/else/_49/Assert/AssertGuard/Assert}}]] [[MultiDeviceIteratorGetNextFromShard]] [[RemoteCall]]
Resolved. For me the problem was that in some cases 'xmin' were larger than 'xmax'.
To my experience this error appears when there is something wrong with the data.
same here
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.408593744]] [[0.384375]] [[{{node Assert/AssertGuard/else/_49/Assert/AssertGuard/Assert}}]] [[MultiDeviceIteratorGetNextFromShard]] [[RemoteCall]]
Resolved. For me the problem was that in some cases 'xmin' were larger than 'xmax'.
To my experience this error appears when there is something wrong with the data.
Thanks, it solved my problem after fixing my annotations.
I got the same error with mask rcnn inception reset. can you please tell how did u fix your annotations?
I got the same error with mask rcnn inception reset. can you please tell how did u fix your annotations?
annotations should be from top left to bottom right. Take a look at video i am sharing below
https://drive.google.com/file/d/1pSLMu6lEqpdvYr2XqaDaEUcVyIp-YEOi/view?usp=sharing
can u tell me the name of software you are using to create the annotation?
one last question is it making mask labels? and have you try it with mask rcnn?
No it is not mask rcnn. It is a normal way of annotating objects inside a dataset.
I haven't tried mask rcnn.
If the same set of annotations works for SSD Mobilenet/ FastRCNNResnet50 then why is there issue with MaskRCNN? In my case training happens for mobilenet/fastrcnn but not for MaskRCNN....
Hello, I have the same issue with mask rcnn. I am using labelme to generate masks and converting json files from label me format to coco format json file. And then using the coco format json file to generate tf records.
I get the same error message.
Has any one resolved it?
the same error,the the error happen in maskrcnn train while no problem in other model like ssd,I use the labelimg the picture to xml,to csv,to record, my xml_csv like this
import os, sys
import glob
import pandas as pd
import xml.etree.ElementTree as ET
def xml_to_csv(_path, _out_file):
xml_list = []
for xml_file in glob.glob(_path + '/*.xml'):
tree = ET.parse(xml_file)
root = tree.getroot()
for member in root.findall('object'):
value = (root.find('filename').text,
int(root.find('size')[0].text),
int(root.find('size')[1].text),
member[0].text,
int(member[4][0].text),
int(member[4][1].text),
int(member[4][2].text),
int(member[4][3].text)
)
xml_list.append(value)
column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']
xml_df = pd.DataFrame(xml_list, columns=column_name)
xml_df.to_csv(_out_file, index=None)
print('Successfully converted xml to csv.')
if __name__ == '__main__':
# convert
xml_to_csv(sys.argv[1], sys.argv[2])
csv_tfrecord.py like this
import os
import io
import pandas as pd
import tensorflow.compat.v1 as tf
from PIL import Image
from object_detection.utils import dataset_util
flags = tf.app.flags
flags.DEFINE_string('csv_input', '', 'Path to the CSV input')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
flags.DEFINE_string('image_path','image/train','Path to the image_dir input')
FLAGS = flags.FLAGS
def class_text_to_int(row_label):
if row_label == 'Jam':
return 1
if row_label == 'Target':
return 2
if row_label == 'Clutter':
return 3
def create_tf_example(row):
#full_path = os.path.join(os.getcwd(), 'images', '{}'.format(row['filename']))
full_path = os.path.join(os.path.dirname(FLAGS.csv_input), FLAGS.image_path, '{}'.format(row['filename']))
with tf.gfile.GFile(full_path, 'rb') as fid:
encoded_jpg = fid.read()
encoded_jpg_io = io.BytesIO(encoded_jpg)
image = Image.open(encoded_jpg_io)
width, height = image.size
filename = row['filename'].encode('utf8')
image_format = b'jpg'
xmins = [row['xmin'] / width]
xmaxs = [row['xmax'] / width]
ymins = [row['ymin'] / height]
ymaxs = [row['ymax'] / height]
classes_text = [row['class'].encode('utf8')]
classes = [class_text_to_int(row['class'])]
tf_example = tf.train.Example(features=tf.train.Features(feature={
'image/height': dataset_util.int64_feature(height),
'image/width': dataset_util.int64_feature(width),
'image/filename': dataset_util.bytes_feature(filename),
'image/source_id': dataset_util.bytes_feature(filename),
'image/encoded': dataset_util.bytes_feature(encoded_jpg),
'image/format': dataset_util.bytes_feature(image_format),
'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example
def main(_):
writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
examples = pd.read_csv(FLAGS.csv_input)
for index, row in examples.iterrows():
tf_example = create_tf_example(row)
writer.write(tf_example.SerializeToString())
writer.close()
if __name__ == '__main__':
tf.app.run()
I also view the record by the scrpit below:
"""
view_records.py:
Consume and display data from a tfrecord file: pulls image and bounding boxes for display
so you can make sure things look reasonabloe, e.g., after augmentation.
Hit 'n' for 'next' image, or 'esc' to quit.
Part of tensorflow-view repo: https://github.com/EricThomson/tfrecord-view
"""
import cv2
import numpy as np
import tensorflow.compat.v1 as tf
tf.enable_eager_execution()
import warnings
warnings.filterwarnings('ignore', category = FutureWarning) #tf 1.14 and np 1.17 are clashing: temporary solution
def cv_bbox(image, bbox, color = (255, 255, 255), line_width = 2):
"""
use opencv to add bbox to an image
assumes bbox is in standard form x1 y1 x2 y2
"""
cv2.rectangle(image, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, line_width)
return
def parse_record(data_record):
"""
parse the data record from a tfrecord file, typically pulled from an iterator,
in this case a one_shot_iterator created from the dataset.
"""
feature = {'image/encoded': tf.FixedLenFeature([], tf.string),
'image/object/class/label': tf.VarLenFeature(tf.int64),
'image/object/bbox/xmin': tf.VarLenFeature(tf.float32),
'image/object/bbox/ymin': tf.VarLenFeature(tf.float32),
'image/object/bbox/xmax': tf.VarLenFeature(tf.float32),
'image/object/bbox/ymax': tf.VarLenFeature(tf.float32),
'image/filename': tf.FixedLenFeature([], tf.string)
}
return tf.parse_single_example(data_record, feature)
def view_records(file_path, class_labels, stride = 1, verbose = 1):
"""
peek at the data using opencv and tensorflow tools.
Inputs:
file_path: path to tfrecord file (usually has 'record' extension)
class_labels: dictionary of labels with name:number pairs (start with 1)
stride (default 1): how many records to jump (you might have thousands so skip a few)
verbose (default 1): display text output if 1, display nothing except images otherwise.
Usage:
Within the image window, enter 'n' for next image, 'esc' to stop seeing images.
"""
dataset = tf.data.TFRecordDataset([file_path])
record_iterator = dataset.make_one_shot_iterator()
num_records = dataset.reduce(np.int64(0), lambda x, _: x + 1).numpy()
if verbose:
print(f"\nGoing through {num_records} records with a stride of {stride}.")
print("Enter 'n' to bring up next image in record.\n")
for im_ind in range(num_records):
#Parse and process example
parsed_example = parse_record(record_iterator.get_next())
if im_ind % stride != 0:
continue
fname = parsed_example['image/filename'].numpy()
encoded_image = parsed_example['image/encoded']
image_np = tf.image.decode_image(encoded_image, channels=3).numpy()
labels = tf.sparse_tensor_to_dense(parsed_example['image/object/class/label'], default_value=0).numpy()
x1norm = tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/xmin'], default_value=0).numpy()
x2norm = tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/xmax'], default_value=0).numpy()
y1norm = tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/ymin'], default_value=0).numpy()
y2norm = tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/ymax'], default_value=0).numpy()
num_bboxes = len(labels)
#% Process and display image
height, width = image_np[:, :, 1].shape
image_copy = image_np.copy()
image_rgb = cv2.cvtColor(image_copy, cv2.COLOR_BGR2RGB)
if num_bboxes > 0:
x1 = np.int64(x1norm*width)
x2 = np.int64(x2norm*width)
y1 = np.int64(y1norm*height)
y2 = np.int64(y2norm*height)
for bbox_ind in range(num_bboxes):
bbox = (x1[bbox_ind], y1[bbox_ind], x2[bbox_ind], y2[bbox_ind])
label_name = list(class_labels.keys())[list(class_labels.values()).index(labels[bbox_ind])]
label_position = (bbox[0] + 5, bbox[1] + 20)
cv_bbox(image_rgb, bbox, color = (250, 250, 150), line_width = 2)
cv2.putText(image_rgb,
label_name,
label_position,
cv2.FONT_HERSHEY_SIMPLEX,
1, (10, 10, 255), 2); #scale, color, thickness
if verbose:
print(f"\nImage {im_ind}")
print(f" {fname}")
print(f" Height/width: {height, width}")
print(f" Num bboxes: {num_bboxes}")
cv2.imshow("bb data", image_rgb)
k = cv2.waitKey()
if k == 27:
break
elif k == ord('n'):
continue
cv2.destroyAllWindows()
if verbose:
print("\n\ntfrecord-view: done going throug the data.")
if __name__ == '__main__':
class_labels = {"Jam" : 1, "Target": 2,"Clutter": 3 }
#Make the following using voc_to_tfr.py
data_path = r"train.record"
verbose = 1
stride = 1
view_records(data_path, class_labels, stride = stride, verbose = verbose)
that look like no problem,and no problem in othet object detect model. have somebody solved the problem?
one last question is it making mask labels? and have you try it with mask rcnn?
have you sloved it, i face the same error in mask_rcnn
Same problem here. Has anyone found a solution yet. It seems there is a compatibility issue in loading the pre-trained checkpoint.
Traceback (most recent call last): File "/Users/_dga/ml-git/tf-ark/Tensorflow/workspace/training_demo/model_main_tf2.py", line 126, in <module> tf.compat.v1.app.run() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/platform/app.py", line 36, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/Users/_dga/ml-git/tf-ark/Tensorflow/workspace/training_demo/model_main_tf2.py", line 117, in main model_lib_v2.train_loop( File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/object_detection/model_lib_v2.py", line 605, in train_loop load_fine_tune_checkpoint( File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/object_detection/model_lib_v2.py", line 401, in load_fine_tune_checkpoint _ensure_model_is_built(model, input_dataset, unpad_groundtruth_tensors) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/object_detection/model_lib_v2.py", line 161, in _ensure_model_is_built features, labels = iter(input_dataset).next() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 260, in next return self.__next__() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 264, in __next__ return self.get_next() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 325, in get_next return self._get_next_no_partial_batch_handling(name) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 361, in _get_next_no_partial_batch_handling replicas.extend(self._iterators[i].get_next_as_list(new_name)) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 1427, in get_next_as_list return self._format_data_list_with_options(self._iterator.get_next()) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py", line 553, in get_next result.append(self._device_iterators[i].get_next()) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 867, in get_next return self._next_internal() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 777, in _next_internal ret = gen_dataset_ops.iterator_get_next( File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3028, in iterator_get_next _ops.raise_from_not_ok_status(e, name) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6656, in raise_from_not_ok_status raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__IteratorGetNext_output_types_18_device_/job:localhost/replica:0/task:0/device:GPU:0}} indices[0] = 0 is not in [0, 0) [[{{node GatherV2_7}}]] [[MultiDeviceIteratorGetNextFromShard]] [[RemoteCall]] [Op:IteratorGetNext] name:
Prerequisites
Please answer the following questions for yourself before submitting an issue.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/using_your_own_dataset.md
2. Describe the bug
I am trying to write the equivalent of this code which converts CSV to TF records but instead, I am trying to convert from JSON to TFrecords. I am trying to generate TFrecords for using it in object detection API.
Here is my full error message
And here is my code, which is an attempt to convert JSON files into TFrecords
Sample JSON file
My Python Code
Can someone give any tips on what is going wrong?
3. Steps to reproduce
I execute the code above to convert JSON to TF records
4. Expected behavior
A TF record that works which I can use when executing the following
Here is my config file
5. Additional context
6. System information