How to train on custom dataset?

[Karthik-Suresh93]

Could someone please enumerate the steps needed to train these models on a custom dataset? I can get my data in the PASCAL VOC format

Thank you very much!

[zik1ng]

I'v trained a R-FCN model using my custom dataset. Steps are as follows. A) MAKE YOUR OWN DATASET IN PASCAL VOC FORMAT

# get the code
git clone https://github.com/msracver/Deformable-ConvNets.git
# make dirs
cd Deformable-ConvNets/
sh init.sh
cd data/
mkdir VOCdevkit && cd VOCdevkit
mkdir VOC2007 && cd VOC2007
mkdir Annotations
mkdir JPEGImages
mkdir ImageSets && cd ImageSets
mkdir Main

Move your .jpg files into JPEGImages folder and .xml files into Annotations folder.

B) SPLIT DATASET INTO TRAINVAL PART AND TEST PART Use Python script as follow.

import os  
import random   

xmlfilepath=r'/your/path/to/VOCdevkit/VOC2007/Annotations'  
saveBasePath=r"/your/path/to/VOCdevkit"  

trainval_percent=0.9   # 90% of data into trainval set
train_percent=0.9       # 90% of data in trainval into train set
total_xml = os.listdir(xmlfilepath)  
num=len(total_xml)    
list=range(num)    
tv=int(num*trainval_percent)    
tr=int(tv*train_percent)    
trainval= random.sample(list,tv)    
train=random.sample(trainval,tr)    

print("train and val size",tv)  
print("traub suze",tr)  
ftrainval = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/trainval.txt'), 'w')    
ftest = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/test.txt'), 'w')    
ftrain = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/train.txt'), 'w')    
fval = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/val.txt'), 'w')    

for i  in list:    
    name=total_xml[i][:-4]+'\n'    
    if i in trainval:    
        ftrainval.write(name)    
        if i in train:    
            ftrain.write(name)    
        else:    
            fval.write(name)    
    else:    
        ftest.write(name)    

ftrainval.close()    
ftrain.close()    
fval.close()    
ftest .close()  

Now you get .txt files in path ~/data/VOCdevkit/VOC2007/ImageSets/Main . Done.

C) MODIFY CONFIG FILES Modify resnet_v1_101_voc0712_rfcn_end2end_ohem.yaml in path ~/experiments/rfcn/cfgs/

# What have written in line 4 is your output path, feel free to change it.
# From line 5 is to set your GPUs
gpus: '0,1'
# From line 51, if your dataset has 4 different labels, just set the NUM_CLASSES as 5 (4+1, one for backgound)
NUM_CLASSES: 5
# From line 54
image_set: 2007_trainval

Modify pascal_voc.py in path ~/lib/dataset/ .

# From line 44, write down your labels in your dataset
self.classes = ['__background__',  # always index 0
                    'label1', 'label2', ..., 'labeln']

D) START TRAINING Download resnet_101 and save it in ~/model/pretrained_model/.

# Training
python experiments/rfcn/rfcn_end2end_train_test.py --cfg experiments/rfcn/cfgs/resnet_v1_101_voc0712_rfcn_end2end_ohem.yaml

---

Maybe that will work for you. Have a nice day :)

[guoqiangqi]

I'v trained a R-FCN model using my custom dataset. Steps are as follows. A) MAKE YOUR OWN DATASET IN PASCAL VOC FORMAT

# get the code
git clone https://github.com/msracver/Deformable-ConvNets.git
# make dirs
cd Deformable-ConvNets/
sh init.sh
cd data/
mkdir VOCdevkit && cd VOCdevkit
mkdir VOC2007 && cd VOC2007
mkdir Annotations
mkdir JPEGImages
mkdir ImageSets && cd ImageSets
mkdir Main

Move your .jpg files into JPEGImages folder and .xml files into Annotations folder.

B) SPLIT DATASET INTO TRAINVAL PART AND TEST PART Use Python script as follow.

import os  
import random   

xmlfilepath=r'/your/path/to/VOCdevkit/VOC2007/Annotations'  
saveBasePath=r"/your/path/to/VOCdevkit"  

trainval_percent=0.9   # 90% of data into trainval set
train_percent=0.9       # 90% of data in trainval into train set
total_xml = os.listdir(xmlfilepath)  
num=len(total_xml)    
list=range(num)    
tv=int(num*trainval_percent)    
tr=int(tv*train_percent)    
trainval= random.sample(list,tv)    
train=random.sample(trainval,tr)    

print("train and val size",tv)  
print("traub suze",tr)  
ftrainval = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/trainval.txt'), 'w')    
ftest = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/test.txt'), 'w')    
ftrain = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/train.txt'), 'w')    
fval = open(os.path.join(saveBasePath,'VOC2007/ImageSets/Main/val.txt'), 'w')    

for i  in list:    
    name=total_xml[i][:-4]+'\n'    
    if i in trainval:    
        ftrainval.write(name)    
        if i in train:    
            ftrain.write(name)    
        else:    
            fval.write(name)    
    else:    
        ftest.write(name)    

ftrainval.close()    
ftrain.close()    
fval.close()    
ftest .close()  

Now you get .txt files in path ~/data/VOCdevkit/VOC2007/ImageSets/Main . Done.

C) MODIFY CONFIG FILES Modify resnet_v1_101_voc0712_rfcn_end2end_ohem.yaml in path ~/experiments/rfcn/cfgs/

# What have written in line 4 is your output path, feel free to change it.
# From line 5 is to set your GPUs
gpus: '0,1'
# From line 51, if your dataset has 4 different labels, just set the NUM_CLASSES as 5 (4+1, one for backgound)
NUM_CLASSES: 5
# From line 54
image_set: 2007_trainval

Modify pascal_voc.py in path ~/lib/dataset/ .

# From line 44, write down your labels in your dataset
self.classes = ['__background__',  # always index 0
                    'label1', 'label2', ..., 'labeln']

D) START TRAINING Download resnet_101 and save it in ~/model/pretrained_model/.

# Training
python experiments/rfcn/rfcn_end2end_train_test.py --cfg experiments/rfcn/cfgs/resnet_v1_101_voc0712_rfcn_end2end_ohem.yaml

Maybe that will work for you. Have a nice day :)

I have a doubt when i trained my model on my costum dataset , i want to know the diffenence betweeen the rfcn_end2end_train_test.py and the rfcn_train_test.py? The other question is what is the script ,rfcn_alternate_train_test.py work for?

[zik1ng]

Sorry that I do not know the differences in these three scripts. Maybe it is due to the different versions, but offer same funtions. You can compare the different scripts by yourself.