search

EdjeElectronics / TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10

How to train a TensorFlow Object Detection Classifier for multiple object detection on Windows
Apache License 2.0
2.92k stars 1.3k forks source link

tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: #225

Open StewartTsai opened 5 years ago

StewartTsai commented 5 years ago

I have problem with my generate_tfrecord.py How can I solve it?

Error below here

Traceback (most recent call last): File "generate_tfrecord.py", line 100, in tf.app.run() File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run _sys.exit(main(argv)) File "generate_tfrecord.py", line 91, in main tf_example = create_tf_example(group, path) File "generate_tfrecord.py", line 46, in create_tf_example encoded_jpg = fid.read() File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 125, in read self._preread_check() File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 85, in _preread_check compat.as_bytes(self.name), 1024 * 512, status) File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 526, in exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: D:\Tensorflow\models\research\object_detection\image\train\1 : \udca8t\udcb2Χ䤣\udca8\udcec\udcab\udcfc\udca9w\udcaa\udcba\udcc0ɮסC ; No such file or directory

I don't know what it means "No such file or directory" But my directory is correct

default

generate_tfrecord.py source code here

import pandas as pd import tensorflow as tf

from PIL import Image from object_detection.utils import dataset_util from collections import namedtuple, OrderedDict

flags = tf.app.flags flags.DEFINE_string('csv_input', '', 'Path to the CSV input') flags.DEFINE_string('image_dir', '', 'Path to the image directory') flags.DEFINE_string('output_path', '', 'Path to output TFRecord') FLAGS = flags.FLAGS

TO-DO replace this with label map def class_text_to_int(row_label): if row_label == 'Bactrocera_dorsalis': return 1 else: None

def split(df, group): data = namedtuple('data', ['filename', 'object']) gb = df.groupby(group) return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path): with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: encoded_jpg = fid.read() encoded_jpg_io = io.BytesIO(encoded_jpg) image = Image.open(encoded_jpg_io) width, height = image.size

filename = group.filename.encode('utf8')
image_format = b'jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []

for index, row in group.object.iterrows():
    xmins.append(row['xmin'] / width)
    xmaxs.append(row['xmax'] / width)
    ymins.append(row['ymin'] / height)
    ymaxs.append(row['ymax'] / height)
    classes_text.append(row['class'].encode('utf8'))
    classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={
    'image/height': dataset_util.int64_feature(height),
    'image/width': dataset_util.int64_feature(width),
    'image/filename': dataset_util.bytes_feature(filename),
    'image/source_id': dataset_util.bytes_feature(filename),
    'image/encoded': dataset_util.bytes_feature(encoded_jpg),
    'image/format': dataset_util.bytes_feature(image_format),
    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
    'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(os.getcwd(), FLAGS.image_dir) examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

if name == 'main': tf.app.run()

srinibujjay commented 5 years ago

i am also facing similar error as below: tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: C:\tensorflow1\models\research\object_detection\images\test\Broken rail013.jpg : The system cannot find the file specified. ; No such file or directory

Dev-HJYoo commented 5 years ago

i think your file'size is small, don't you?

srinibujjay commented 5 years ago

The error is rectified by matching jpg file and xml file total count (one xml file for each jpg file) For me the issue is closed now

sumannelli commented 5 years ago

I am also facing the same issue. I checked the quantities of images and .xml files both having the equal count. Provided the valid path. Please help me.

AyazYousafxai commented 5 years ago

make sure in xml file the filename extension is exist or not like

train 
<filename>**cam_image1.jpg**</filename>
AyazYousafxai commented 5 years ago
train 
<filename>cam_image1.jpg</filename>
ChamodyaDias commented 5 years ago

Check your train_labels.csv & test_labels.csv files and check the "Filename" column. It should have the name of the image as "yourImgName.png / yourImgName.jgp" etc. The extension should be there. Unless, change the code of "xml_to_csv.py" as follows: 

def xml_to_csv(path):
    xml_list = []
    for xml_file in glob.glob(path + '/*.xml'):
        tree = ET.parse(xml_file)
        root = tree.getroot()
        for member in root.findall('object'):
            value = (root.find('filename').text + '.png', #Put your file extension here
                     int(root.find('size')[0].text),
                     int(root.find('size')[1].text),
                     member[0].text,
                     int(member[4][0].text),
                     int(member[4][1].text),
                     int(member[4][2].text),
                     int(member[4][3].text)
                     )
            xml_list.append(value)
    column_name = ['filename', 'width', 'height',
               'class', 'xmin', 'ymin', 'xmax', 'ymax']
    xml_df = pd.DataFrame(xml_list, columns=column_name)
    return xml_df
daniablank commented 5 years ago

My CSV has the extensions in it, and I can verify that the image files that it is claiming do not exist, do exist. I can copy the exact filepath in the error and check if it is a file, and it is. In another instance of this issue, I've been told that this error is thrown if the boxes are under 33 pixels on a side, but that isn't the case for the image it is failing to detect. Are there other conditions apart from the filepath not leading to an actual file that cause this error?

radheyasarode commented 5 years ago

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

solomonvimal commented 4 years ago

I have the same issue - but my files are not in .jpg format. Do .tif files work in tensorflow?

muhammadaamir123 commented 4 years ago

When I execute this command: python3 train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config

WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

WARNING:tensorflow:From train.py:55: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From train.py:55: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From train.py:167: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From train.py:89: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W1212 03:17:10.382213 6232 deprecation_wrapper.py:119] From train.py:89: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\utils\config_util.py:86: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W1212 03:17:10.384212 6232 deprecation_wrapper.py:119] From c:\users\aamir\desktop\models\research\object_detection\utils\config_util.py:86: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From train.py:94: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

W1212 03:17:10.388208 6232 deprecation_wrapper.py:119] From train.py:94: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\trainer.py:228: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.create_global_step W1212 03:17:10.411310 6232 deprecation.py:323] From c:\users\aamir\desktop\models\research\object_detection\trainer.py:228: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.create_global_step WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:104: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W1212 03:17:10.418304 6232 deprecation_wrapper.py:119] From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:104: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:119: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W1212 03:17:10.419303 6232 deprecation_wrapper.py:119] From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:119: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

Traceback (most recent call last): File "train.py", line 167, in tf.app.run() File "C:\Users\Aamir\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\Aamir\Anaconda3\lib\site-packages\absl\app.py", line 299, in run _run_main(main, args) File "C:\Users\Aamir\Anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main sys.exit(main(argv)) File "train.py", line 163, in main worker_job_name, is_chief, FLAGS.train_dir) File "c:\users\aamir\desktop\models\research\object_detection\trainer.py", line 235, in train train_config.prefetch_queue_capacity, data_augmentation_options) File "c:\users\aamir\desktop\models\research\object_detection\trainer.py", line 59, in create_input_queue tensor_dict = create_tensor_dict_fn() File "train.py", line 120, in get_next dataset_builder.build(config)).get_next() File "c:\users\aamir\desktop\models\research\object_detection\builders\dataset_builder.py", line 138, in build label_map_proto_file=label_map_proto_file) File "c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py", line 195, in init use_display_name) File "c:\users\aamir\desktop\models\research\object_detection\utils\label_map_util.py", line 149, in get_label_map_dict label_map = load_labelmap(label_map_path) File "c:\users\aamir\desktop\models\research\object_detection\utils\label_map_util.py", line 129, in load_labelmap label_map_string = fid.read() File "C:\Users\Aamir\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 122, in read self._preread_check() File "C:\Users\Aamir\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 84, in _preread_check compat.as_bytes(self.__name), 1024 * 512) tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: data/object-detection.pbtxt : The system cannot find the file specified. ; No such file or directory

Any solution?

SUFEHeisenberg commented 4 years ago

tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: : ϵͳ\udcd5Ҳ\udcbb\udcb5\u dcbdָ\udcb6\udca8\udcb5\udcc4·\udcbe\udcb6\udca1\udca3 ; No such process I met with the same questions yesterday which confused me all day along 0.0 But today I successed with two changes:

  1. Change all path into GetFullPath forms
  2. delete the 'space' between the equal and path e.g. --input_file='F:/PycharmProjects/data/tf_train.tfrecord' And then made it. Dunno the reason but you can have a try.
ajeeth-b commented 4 years ago

Your csv file have all the images name under filename column without its type i.e extension, you have to add the format of all image under the filename column in csv file.

harshita897 commented 4 years ago

error give an alternative for this problem

JustinsDevAccount commented 4 years ago

What helped me in this scenario is to specify the full path, ex. C:\Users\justi\Deep_Learning\Object_Recognition\research\object_detection\images\train INSTEAD OF: images\train Also, keep an eye on if your folder specifiers are '/' or '\' depending on your OS.

ajithvcoder commented 4 years ago

try to print the path and see

def create_tf_example(group, path):
    print(path)
    print(os.path.join(path, '{}'.format(group.filename)))

whether the path is correct or not

arunkumar27karthi commented 4 years ago

whether the code is correct or not

''' Usage:

From tensorflow/models/

Create train data:

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

Create test data:

python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=test.record ''' from future import division from future import print_function from future import absolute_import

import os import io import pandas as pd import tensorflow.compat.v1 as tf

from PIL import Image from object_detection.utils import dataset_util from collections import namedtuple, OrderedDict

flags = tf.app.flags flags.DEFINE_string('csv_input', '', 'Path to the CSV input') flags.DEFINE_string('output_path', '', 'Path to output TFRecord') flags.DEFINE_string('image_dir', '', 'Path to images') FLAGS = flags.FLAGS

TO-DO replace this with label map

def class_text_to_int(row_label): if row_label == 'men': return 1 elif row_label == 'women': return 2 elif row_label == 'cat': return 3 else: None

def split(df, group): data = namedtuple('data', ['filename', 'object']) gb = df.groupby(group) return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path): with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: encoded_jpg = fid.read() encoded_jpg_io = io.BytesIO(encoded_jpg) image = Image.open(encoded_jpg_io) width, height = image.size

filename = group.filename.encode('utf8')
image_format = b'jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []

for index, row in group.object.iterrows():
    xmins.append(row['xmin'] / width)
    xmaxs.append(row['xmax'] / width)
    ymins.append(row['ymin'] / height)
    ymaxs.append(row['ymax'] / height)
    classes_text.append(row['class'].encode('utf8'))
    classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={
    'image/height': dataset_util.int64_feature(height),
    'image/width': dataset_util.int64_feature(width),
    'image/filename': dataset_util.bytes_feature(filename),
    'image/source_id': dataset_util.bytes_feature(filename),
    'image/encoded': dataset_util.bytes_feature(encoded_jpg),
    'image/format': dataset_util.bytes_feature(image_format),
    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
    'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(FLAGS.image_dir) examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

if name == 'main': tf.app.run()

ajithvcoder commented 4 years ago

@arunkumar27karthi what is the output u are getting ? any erro

arunkumar27karthi commented 4 years ago

i got this error for the above code

2020-03-08 21:22:29.141659: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll Traceback (most recent call last): File "generate_tf_records.py", line 104, in tf.app.run() File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 299, in run _run_main(main, args) File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 250, in _run_main sys.exit(main(argv)) File "generate_tf_records.py", line 95, in main tf_example = create_tf_example(group, path) File "generate_tf_records.py", line 50, in create_tf_example encoded_jpg = fid.read() File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 122, in read self._preread_check() File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 84, in _preread_check compat.as_bytes(self.__name), 1024 * 512) tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: images/train\cat1.jpg : The system cannot find the path specified. ; No such process

arunkumar27karthi commented 4 years ago

is i should download the racoon dataset and work on it ?

ajithvcoder commented 4 years ago

u can use any dataset . see this path . NewRandomAccessFile failed to Create/Open: images/train\cat1.jpg . its wrong .

give the path manually in below mentioned path variable

def main(_):
writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
path = os.path.join(FLAGS.image_dir)

you can check but adding print in below function

def create_tf_example(group, path):
    print(path)
    print(os.path.join(path, '{}'.format(group.filename)))
shivprasad94 commented 4 years ago

i had same issue, but this had something to do with my images/train folder and respective xml files miss match and label.csv

even if there is small issue/ miss match in images and xml you will get error...

step 1 : check - image/train/cat1.jpg step 2 : check the file name col data in your- train_labels.csv the name should be exactly matching in images/train and train_labels.csv (verify with all images,xml)

once everything is done run below command..

python generate_tfrecord.py --csv_input=images/train_labels.csv --image_dir=images/train --output_path=train.record

niraj3819 commented 4 years ago

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

i tried but it didn't worked for me. can you please suggest some new method.

arunkumar27karthi commented 4 years ago

Can I recognise object using tensorflow from matplotlib 3d view

On Tue, 28 Apr 2020, 5:39 pm niraj3819, notifications@github.com wrote:

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

i tried but it didn't worked for me. can you please suggest some new method.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10/issues/225#issuecomment-620565341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANORDGLDIAHJ7736BP6LWNTRO3BPNANCNFSM4GUAWCZA .

CHERYZ commented 4 years ago

Hi, I solved this question just modified generat_tfrecord.py code like this: image Change the path to your own file path. And then just run the generate_tfrecord.py

SrihariKalyanaraman commented 4 years ago

I got this error and I made a small change in the generate_tfRecords.py code

While mentioning the path in main function instead of images give '/images/train ' while generating tfRecords for Train data and give '/images/test ' while generating tfRecords for test data

Few Warnings might come due to tensorflow version compatibility , but the records are generated successfully.

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(os.getcwd(), 'images/train/') # I have changed in this line examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

tf-main-github

Armand-Eicker-Dev commented 3 years ago

@SrihariKalyanaraman i was searching this thread looking for answers, the fix you posted above SOLVED my problem. Just wanted to say thanks...sometimes the fix can be right under our noses.

ljubantomic01 commented 3 years ago

I got this error and I made a small change in the generate_tfRecords.py code

While mentioning the path in main function instead of images give '/images/train ' while generating tfRecords for Train data and give '/images/test ' while generating tfRecords for test data

Few Warnings might come due to tensorflow version compatibility , but the records are generated successfully.

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(os.getcwd(), 'images/train/') # I have changed in this line examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

tf-main-github

Thank you very much!!!

Petros626 commented 3 years ago

try to print the path and see

def create_tf_example(group, path):
    print(path)
    print(os.path.join(path, '{}'.format(group.filename)))

whether the path is correct or not

where i must run these commands? in anaconda prompt as admin?

alwaysaditi commented 2 years ago

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

thanks a lot for this!it worked for me

alwaysaditi commented 2 years ago

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

thank you so much!