"Path does not exist" while training my own dataset

[movie3105]

Set proposal method: gt Appending horizontally-flipped training examples... Traceback (most recent call last): File "train.py", line 215, in train = Train() File "train.py", line 69, in init self.imdb, self.roidb = combined_roidb("voc_2007_trainval") File "train.py", line 48, in combined_roidb roidbs = [get_roidb(s) for s in imdb_names.split('+')] File "train.py", line 48, in roidbs = [get_roidb(s) for s in imdb_names.split('+')] File "train.py", line 45, in get_roidb roidb = get_training_roidb(imdb) File "train.py", line 25, in get_training_roidb imdb.append_flipped_images() File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\imdb.py", line 112, in append_flipped_images widths = self._get_widths() File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\imdb.py", line 108, in _get_widths for i in range(self.num_images)] File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\imdb.py", line 108, in for i in range(self.num_images)] File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\pascal_voc.py", line 63, in image_path_at return self.image_path_from_index(self._image_index[i]) File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\pascal_voc.py", line 72, in image_path_from_index 'Path does not exist: {}'.format(image_path) AssertionError: Path does not exist: E:\Program\Faster-RCNN-TensorFlow-Python3-master\data\VOCdevkit2007\VOC2007\JPEGImages\000005.jpg

@morpheusthewhite why this error occured ? i already delete all the former training data a at E:\Program\Faster-RCNN-TensorFlow-Python3-master\data\VOCdevkit2007\VOC2007\JPEGImages and change it to my own dataset but why it searching for the old dataset ?

Originally posted by @movie3105 in https://github.com/dBeker/Faster-RCNN-TensorFlow-Python3/issues/82#issuecomment-512147525

[movie3105]

this is my xml structure

[S0soo]

You need to convert your dataset into VOC dataset

[movie3105]

@S0soo can you tell me how to do that? or do you have a source of tutorial about how to do that ? thank you

[S0soo]

@S0soo can you tell me how to do that? or do you have a source of tutorial about how to do that ? thank you

He gets the image index by reading the file trainval.txt. You need to check your .XML file, whether the name of the image file corresponds to it or not. My GitHub library has a script for converting the VOC dataset. If you need to, you can try to use it.

[movie3105]

ill try it for now @S0soo thank you

[S0soo]

You're welcome. Good luck.

[movie3105]

but where i run the program to make the right directory ? is this suppose to be like this ? @S0soo

[S0soo]

You can run it anywhere. It's just a script file.

[movie3105]

why it didnt do anything @S0soo ? it didnt make a proggress

[S0soo]

The first line of input is the path to your image The second line of input is the path of your label The third line of input is that you save the transformed data set path

[movie3105]

owh thank you ill try it again @S0soo

[movie3105]

E:\Program\Faster-RCNN-TensorFlow-Python3-master>python train.py Loaded dataset voc_2007_trainval for training Set proposal method: gt Appending horizontally-flipped training examples... Traceback (most recent call last): File "train.py", line 215, in train = Train() File "train.py", line 69, in init self.imdb, self.roidb = combined_roidb("voc_2007_trainval") File "train.py", line 48, in combined_roidb roidbs = [get_roidb(s) for s in imdb_names.split('+')] File "train.py", line 48, in roidbs = [get_roidb(s) for s in imdb_names.split('+')] File "train.py", line 45, in get_roidb roidb = get_training_roidb(imdb) File "train.py", line 25, in get_training_roidb imdb.append_flipped_images() File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\imdb.py", line 114, in append_flipped_images boxes = self.roidb[i]['boxes'].copy() File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\imdb.py", line 75, in roidb self._roidb = self.roidb_handler() File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\pascal_voc.py", line 112, in gt_roidb for index in self.image_index] File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\pascal_voc.py", line 112, in for index in self.image_index] File "E:\Program\Faster-RCNN-TensorFlow-Python3-master\lib\datasets\pascal_voc.py", line 170, in _load_pascal_annotation cls = self._class_to_ind[obj.find('name').text.lower().strip()] KeyError: 'granny_smith'

so i already done with the trainval but this error occured what does this mean ? @S0soo can you help me ?

[movie3105]

path of your labebl

The first line of input is the path to your image The second line of input is the path of your label The third line of input is that you save the transformed data set path

so the path of my label means my .xml file right ?

[S0soo]

path of your labebl

The first line of input is the path to your image The second line of input is the path of your label The third line of input is that you save the transformed data set path

so the path of my label means my .xml file right ?

yes

[S0soo]

granny_smith

granny_smith Is it one of your class? Maybe it's capitalized on your label. try to change “cls = self._class_to_ind[obj.find('name').text.lower().strip()]” to “cls = self._class_to_ind[obj.find('name').text.strip()]”

[S0soo]

granny_smith

granny_smith Is it one of your class? Maybe it's capitalized on your label. try to change “cls = self._class_to_ind[obj.find('name').text.lower().strip()]” to “cls = self._class_to_ind[obj.find('name').text.strip()]”

It's also possible that your XML contains granny_smith, but you don't use it.

[movie3105]

granny_smith

granny_smith Is it one of your class? Maybe it's capitalized on your label. try to change “cls = self._class_to_ind[obj.find('name').text.lower().strip()]” to “cls = self._class_to_ind[obj.find('name').text.strip()]”

so how can we make the class ? i just make the xml label with LBLIMG and get the xml from that like this is that the class that you mean ? or i have to make a class.txt somewhere

[S0soo]

yes. In XML it is called ”Granny_Smith”, but "cls = self._class_to_ind [obj.find('name').text.lower().strip()]" turn capital letter into small letter，Will become “granny_smith”，It won't find “granny_smith” in your classes. So It makes “KeyError: 'granny_smith'” error.

[S0soo]

at first，In./lib/dataset/pascal_voc.py, you have written your classes name? like this

[movie3105]

Owh i see i didint add my class there now ill add it and run the train again,thank you

[S0soo]

Owh i see i didint add my class there now ill add it and run the train again,thank you

Good luck.

[movie3105]

Sir i try to train, it works but it have 40000 itteration so it takes really long, can we speed up this process or make a check point ? Does the restart for 0 itteration again if i cancel it?

[S0soo]

You can set your checkpoint in the 'snapshot_iterations' parameter in ./lib/config/config.py.

[S0soo]

and i have not tried to continue training at the checkpoint, I think this should be feasible.

[movie3105]

i already start the training a while ago and the checkpoint is 5000 but i change it to 1000 when its running does this will take the snapshot at 1000 or 5000? the checkpoint is that right ?

[S0soo]

I have not tried to modify the snapshot_iterations during training, but I think it will become 1000，if iter == snapshot_iterations, Generate checkpoint at ./default/voc_2007_trainval/default/ like this

[movie3105]

Does using mobilenet faster the training ? Sadly the checkpoint still 5000 Can you tell me the spec of computer that you use to train this ? Or can we change the number of iteration ? sorry if this sounds stupid cause i didnt really know what is iteration

[S0soo]

No, I just replaced vgg16 with mobilenet. my GPU is RTX2080ti of course, you can change the number of iterations. in ./lib/config/config.py, parameter “max_iters”. You can set a smaller number of times, like 10000. meanwhile, You should change the parameters“step_size”, Set a smaller number than "max_iters", like 7000. As for iteration, You can simply understand it as a learning process. Just like your teacher taught you the pronunciation of a new word, he will constantly correct your wrong pronunciation and move towards the correct pronunciation. finally, I suggest you need to know CNN and BP neural network.

[S0soo]

reducing the number of iterations may result in underfitting. this will result in loss of accuracy.

[movie3105]

so how long it takes to train this project at your computer ?

[S0soo]

Usually two and a half hours

[movie3105]

ahahahah the truth is i guess what i need first is a good pc cause i try to train this in a laptop with only Geforce 940MX but thanks for everything and i think i learn much from you @S0soo thank you very much

but this train session running in gpu not cpu right ?

[S0soo]

You're welcome. if you're going to do some deep learning work, you need have a high-performance gpu. Finally, wish you success

[movie3105]

You're welcome. if you're going to do some deep learning work, you need have a high-performance gpu. Finally, wish you success

can we do the train in cpu cause i think i can afford a ram ?

[S0soo]

Yes, but it trains much more slowly than the gpu, and almost no one trains with the CPU

[movie3105]

Yes, but it trains much more slowly than the gpu, and almost no one trains with the CPU

owh sucks but thank you sir wish you all the best

[S0soo]

You can use cloud gpu services, but most of them are not free. I used Floydhub when I didn't have a gpu, and it offered several hours of free gpu. But now I've forgotten how to use it

[movie3105]

path of your labebl

The first line of input is the path to your image The second line of input is the path of your label The third line of input is that you save the transformed data set path

so the path of my label means my .xml file right ?

yes

@S0soo does the result of this program change like this ? and there is no trainval in segementation folder and it missing layout folder isnt it suppose to be like this this the first time i run it and i tru it for second time but result change like picture above

[S0soo]

trainval.txt in Main folder

[S0soo]

if you are doing object detection for images, you don't need the layout and action folders.

[S0soo]

This is the content of voc2012, this script is for making voc2007

[movie3105]

but why when i first try this program it makes layout and segmentation folder with trainval,etc in that

[S0soo]

i m sorry i don't understand what you mean. "this program" mean what,

[movie3105]

i m sorry i don't understand what you mean. "this program" mean what,

thats mean the make-voc2007-database-master that you suggest me to use a week ago but actually it still running without the layout and segmentation folder in there

[movie3105]

im sorry if im asking too much question i just want to make sure the format right before i use cloud gpu to train the dataset but does checkpoint always this big ? i mean i make the checkpoint every 100 iterataion and each checkpoint take 1.0Gb is that really normal ?

[S0soo]

you don't need layout and segmentation for this demo. Just make sure there is trainval.txt in the main folder.

[S0soo]

yes，with vgg16 there will be about 1G of memory, if use resnet50, it makes about 200M, use mobilenet, it makes about 40M. i found that the accuracy of vgg16 is similar to that of resnet50. their accuracy is higher than mobilenet

[movie3105]

so can i delete checkpoint vgg16_faster_rcnn_iter_100 when vgg16_faster_rcnn_iter_200 already made doesnt in vgg16_faster_rcnn_iter_200 already have information about vgg16_faster_rcnn_iter_100 ?

[S0soo]

of course, each checkpoint is independent, you can use any checkpoint to perform the next test. to be exact, the information (weight) contained in each checkpoint is different. you only need to keep the last checkpoint.

[movie3105]

thank you sir very helpful @S0soo