Generation of our own dataset

[eravallirao]

How can we generate the label file for each image? Can someone help me.

[sainatarajan]

I think: You have a set of images and a set of labels for those images. The labels are just the masks of those images. The area of the image which you feel the model should be trained on must be in white and other areas black. I trained my model with this approach, but in the end, when the masks were generated by the model for the test data, it was full black. I don't know where I am going wrong. Any help from someone who was successful would be great.

[sudhasubramaniam]

Hey did you got any solution for this, I'm also facing same issue.

[sainatarajan]

I have obtained the solution. All you have to do is to add the BatchNormalization after every Conv2D layer except for the last Conv2D layer. Ex:

conv1 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inputs)
conv1 = BatchNormalization()(conv1)

Go on till 9th layer. Don't add for 10th Conv2D layer. And also make sure that the resolution of your training set and test set images are divided by 32. Ex. 512x512 or 640x960. Otherwise, you can expect the model to output a blank image.

[sudhasubramaniam]

Thanks a lot. Still I'm facing same issue. Code s available in the link below https://github.com/sudhasubramaniam/segmentation

[sainatarajan]

Thanks a lot. Still I'm facing same issue. Code s available in the link below https://github.com/sudhasubramaniam/segmentation

You need to add the BatchNormalization to the up-sampling layers as well. For example:

up6 = Conv2D(512, 2, activation='relu', padding='same', kernel_initializer='he_normal')(UpSampling2D(size=(2, 2))(drop5))
up6 = BatchNormalization()(up6)

Do this for up6, up7, up8 and up9.

[sudhasubramaniam]

No change. Still facing same issue.

[sainatarajan]

No change. Still facing same issue.

Try the code from my repo. Link here.

[sudhasubramaniam]

Thankyou, It working now. If you see my github repository my training dataset has training image and label. Please let me know whether I can extract boundary information as in label dataset.

[sainatarajan]

Try to keep only the boundary information on the labels. Blackout all areas except for the boundary region so that the network focuses on the specific region. Ex : without those red and yellow lines and then train the network.

[sudhasubramaniam]

Thank you. Let me try as such.

[sudhasubramaniam]

Hi Sai, I trained with boundary information but still my perdiction is not good. (epoch-300). Please comment on causes https://github.com/sudhasubramaniam/segmentation

I used your code for training.

[sainatarajan]

Hi, The possible reasons why your model has failed are:

1. The image set has extra information. Can you remove all those? Like Length, LCCA, + Markers and other such information.

2. The masks are somewhat not clear. There are shades of the surface here and there. The algorithm can make mistakes while learning. Have you done the masks manually? I would suggest you define the Region of Interest (RoI) as a formula or something and then create masks programmatically using OpenCV. In my case, the RoI was a set of points. So, I would create a polygon out of those points and then prepare the masks accordingly.

3. Most importantly: Training Data size. After taking a look at your repository, I found that you were having just 12 images which are very insufficient for training a complicated algorithm like U-Net or any other such algorithm. I had a total of 120 images and out of which 100 were given as training set and the rest 20 were given as test set. I ran the algorithm first for 20 epochs and then for 50 epochs. The accuracy improved. I am not talking about the accuracy that is getting produced as output after every epoch. By accuracy, I mean manually comparing the original test masks with the generated test masks. The generated ones were a little better. Now I might train my U-Net with a set of 1000 images. I also advise you to do the same. A large training set is important.

4. Also, I feel that there might be a case of overfitting the model when you run for 300 epochs. Start with 20 or 25. Then go to 50 and so on.

5. Try to change the metric to IoU (Intersection over Union) in the U-Net code. Might perform a little better.

On Thu, Aug 29, 2019 at 3:50 PM sudhasubramaniam notifications@github.com wrote:

Hi Sai, I trained with boundary information but still my perdiction is not good. (epoch-300). Please comment on causes https://github.com/sudhasubramaniam/segmentation

I used your code for training.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zhixuhao/unet/issues/138?email_source=notifications&email_token=AFPLBSL56LGWFLGVQQ2KIJTQG6PILA5CNFSM4IJ7EKD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5OAD4A#issuecomment-526123504, or mute the thread https://github.com/notifications/unsubscribe-auth/AFPLBSM6X7YTWSYRDU2P4LTQG6PILANCNFSM4IJ7EKDQ .

[sudhasubramaniam]

Thank you for your comments.

[BriFuture]

Try to keep only the boundary information on the labels. Blackout all areas except for the boundary region so that the network focuses on the specific region. Ex : without those red and yellow lines and then train the network.

HI, i'm wondering how to make the label or exactly the mask of the image.

With labelme I get the json file that description different area with different labels, so if I want to export the label as Image, shall I use GIMPS2 to set background as BLACK?

By the way, even if the background can be picked out, how should I notate other area with more than 2 categories (in other words, there are background area, target area, and some fault area).

How can the image be colored by different area and same category should be the same color, like the video shows:

[sainatarajan]

@BriFuture I will tell you the simple steps to create the masks. Assuming you have 2 classes:

1. Open the JSON file in python and process image by image.
2. Read the corresponding image from the disk.
3. Create a mask having the same size as the image using np.zeros()
4. I have not used labelme, but I have used VGG Annotation tool. Here you need to read the coordinates one by one for the corresponding image and then use cv2.drawContours() to fill that polygon with (255, 255, 255) on the mask. Do this for all the polygons.
5. Save the masks separately in a masks folder using cv2.imwrite().
6. Do this for all the images.

[BriFuture]

Thank you. But should different classes be filled with different color? Em, I have tried to train the network with custom datasets (build with GIMP but poorly distinguished), here is a sample:

origin: label:

In the origin image, the white area is one class, while red area is another one, the gray is background. I just colored background with black color. Others are left unchanged. Is that OK?

Do I need leave contours only without any other color information ?

[BriFuture]

I searched google and find this repo which contains 4 classes.

[sainatarajan]

One color for one class. The regions that are not of interest make it black. And also make sure you change the loss function from binary_crossentropy if you are dealing with more than 2 classes.

[sainatarajan]

I searched google and find this repo which contains 4 classes.

Ok, so you can also color your labels(masks) in this way (1, 1, 1) or (2, 2, 2) instead of red, blue, green white etc.

[BriFuture]

thanks for the details. I'll give it a try.

[synioe]

@BriFuture I will tell you the simple steps to create the masks. Assuming you have 2 classes:

1. Open the JSON file in python and process image by image.
2. Read the corresponding image from the disk.
3. Create a mask having the same size as the image using np.zeros()
4. I have not used labelme, but I have used VGG Annotation tool. Here you need to read the coordinates one by one for the corresponding image and then use cv2.drawContours() to fill that polygon with (255, 255, 255) on the mask. Do this for all the polygons.
5. Save the masks separately in a masks folder using cv2.imwrite().
6. Do this for all the images.

Could you please release the code which use to generate masks(labels) with the annotated json file?

[sainatarajan]

@synioe That completely depends on the json file that you have. Different annotation tool generates jsons in a different way.

[synioe]

@sainatarajan I use VGG image annotator to aquire the json file.

[BriFuture]

@synioe
As @sainatarajan said, different tools generates json may differ from eachother. The json format you currently used may be different from mine. Here is a sample code I write to process with labelme:

for ld in sorted(labeldir.iterdir()):
    if ld.suffix == ".json":
        name = str(ld)[:-5]
        print("Processing", name)
        img = cv2.imread(f"{name}.bmp")

        contoursImg = np.zeros(img.shape)
        with open(ld, "r") as f:
            jsoncontent = json.load(f)
            shapes = jsoncontent["shapes"][0]["points"]

            for i, s in enumerate(shapes):
                 shapes[i] = np.array([ np.array(s).astype(int) ])
            contour = np.array(shapes)

            contouredImg = cv2.drawContours(contoursImg, [contour], -1, (255, 255, 0), 2)
            plt.figure()
            plt.imshow(contouredImg)
            plt.show()
            slashpos = name.find('/')
            cv2.imwrite(f'label/{name[slashpos+1:]}.png', contouredImg)

just be aware of that, contours should be colored, while other areas not. By the way, the param contour of drawContours method is hard to understand and construct, I tried several times to get it work.

[sainatarajan]

@synioe Even I use the VGG tool. So in that case I would be able to help you.

[synioe]

@BriFuture Thank you for your sharing. @sainatarajan Thank you all the same.

[sainatarajan]

@synioe This might look a bit complicated, but it might help. Take the delaunay2D file from here

"""
my format of json

{
"image1.jpg":{
        "filename":"image1.jpg",
        "size":123456,
        "regions":[
            {
                "shape_attributes":{
                    "all_points_x":[
                        675,
                        808,
                        957,
                        967,
                        929,
                        791,
                        678,
                        703
                    ],
                    "all_points_y":[
                        543,
                        518,
                        492,
                        722,
                        760,
                        760,
                        760,
                        647
                    ],
                    "name":"polygon"
                },
                "region_attributes":{
                    "Object":"class1",
                }
            },...all other regions..{}]
    }, ..all other images..., {}
}

"""

import glob, os, cv2
import numpy as np
import multiprocessing as mp
from PIL import Image
from delaunay2D import Delaunay2D

def calculateDelauneyPoints(points):
    points= np.array(points).astype(int)
    points= [list(item) for item in points]
    dt = Delaunay2D()
    for s in points:
        dt.addPoint(s)
    coord, tris= dt.exportDT()
    return np.array(coord)

def maskImage(filename, polygons, o_list):
    root_path= '../data/'
    image = cv.imread(root_path+"all_images/"+filename)
    mask = np.zeros(shape = image.shape, dtype = "uint8")
    i= 0
    for points in polygons:
        points= [(int(x), int(y)) for x,y in points]
        try:
            rect= calculateDelauneyPoints(points)
        except:
            continue
        if o_list[i] in ['class1']:
            cv.drawContours(mask,[rect], -1,(1, 1, 1),cv.FILLED)
        elif o_list[i] in ['class2']:
            cv.drawContours(mask,[rect], -1,(2, 2, 2),cv.FILLED)
        elif o_list[i] in ['class3']:
            cv.drawContours(mask,[rect], -1,(3, 3, 3),cv.FILLED)
        i+= 1

    cv.imwrite(root_path+"dataset/images/"+os.path.splitext(filename)[0]+".png", image)
    cv.imwrite(root_path+"dataset/masks/"+os.path.splitext(filename)[0]+".png", mask)

def getRegionProperties(region):
    shape_attributes= region["shape_attributes"]
    region_attributes= region["region_attributes"]
    objects= region_attributes["Object"]
    regions= ['class1', 'class2', 'class3']
    all_points_x= shape_attributes["all_points_x"]
    all_points_y= shape_attributes["all_points_y"]
    coordinates= []
    for i in range(0, len(all_points_x)):
        coordinates.append((all_points_x[i], all_points_y[i]))

    return (objects, coordinates)

def parallelizePlotting(data, region_mappings, json_data):

    polygon_coordinates= {}
    polygons= []
    img_json_data= json_data[data]

    filename= img_json_data["filename"]

    # Open the original image here
    image_matrix= np.array(Image.open('../data/all_images/'+filename), dtype= np.uint8)
    region_data= img_json_data["regions"]
    objects_list= []

    for region in region_data:
        objects, coordinates= getRegionProperties(region)
        if coordinates is not None:
            polygons.append(coordinates)
            polygon_coordinates[objects]= coordinates
            objects_list.append(objects)

    # Masking the images
    maskImage(filename, polygons, objects_list)
    return (filename, polygon_coordinates)

def main():

    root_path= 'path'

    json_file= open(root_path+'filename.json')
    json_data= json.load(json_file)

    # for multiprocessing
    pool = mp.Pool(mp.cpu_count()-1)

    def resultCallback(item):
        return

    for data in json_data:
        pool.apply_async(parallelizePlotting, args=(data, json_data), callback= resultCallback)

    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

[synioe]

@synioe This might look a bit complicated, but it might help. Take the delaunay2D file from here

"""
my format of json

{
"image1.jpg":{
        "filename":"image1.jpg",
        "size":123456,
        "regions":[
            {
                "shape_attributes":{
                    "all_points_x":[
                        675,
                        808,
                        957,
                        967,
                        929,
                        791,
                        678,
                        703
                    ],
                    "all_points_y":[
                        543,
                        518,
                        492,
                        722,
                        760,
                        760,
                        760,
                        647
                    ],
                    "name":"polygon"
                },
                "region_attributes":{
                    "Object":"class1",
                }
            },...all other regions..{}]
    }, ..all other images..., {}
}

"""

import glob, os, cv2
import numpy as np
import multiprocessing as mp
from PIL import Image
from delaunay2D import Delaunay2D

def calculateDelauneyPoints(points):
    points= np.array(points).astype(int)
    points= [list(item) for item in points]
    dt = Delaunay2D()
    for s in points:
        dt.addPoint(s)
    coord, tris= dt.exportDT()
    return np.array(coord)

def maskImage(filename, polygons, o_list):
    root_path= '../data/'
    image = cv.imread(root_path+"all_images/"+filename)
    mask = np.zeros(shape = image.shape, dtype = "uint8")
    i= 0
    for points in polygons:
        points= [(int(x), int(y)) for x,y in points]
        try:
            rect= calculateDelauneyPoints(points)
        except:
            continue
        if o_list[i] in ['class1']:
            cv.drawContours(mask,[rect], -1,(1, 1, 1),cv.FILLED)
        elif o_list[i] in ['class2']:
            cv.drawContours(mask,[rect], -1,(2, 2, 2),cv.FILLED)
        elif o_list[i] in ['class3']:
            cv.drawContours(mask,[rect], -1,(3, 3, 3),cv.FILLED)
        i+= 1

    cv.imwrite(root_path+"dataset/images/"+os.path.splitext(filename)[0]+".png", image)
    cv.imwrite(root_path+"dataset/masks/"+os.path.splitext(filename)[0]+".png", mask)

def getRegionProperties(region):
    shape_attributes= region["shape_attributes"]
    region_attributes= region["region_attributes"]
    objects= region_attributes["Object"]
    regions= ['class1', 'class2', 'class3']
    all_points_x= shape_attributes["all_points_x"]
    all_points_y= shape_attributes["all_points_y"]
    coordinates= []
    for i in range(0, len(all_points_x)):
        coordinates.append((all_points_x[i], all_points_y[i]))

    return (objects, coordinates)

def parallelizePlotting(data, region_mappings, json_data):

    polygon_coordinates= {}
    polygons= []
    img_json_data= json_data[data]

    filename= img_json_data["filename"]

    # Open the original image here
    image_matrix= np.array(Image.open('../data/all_images/'+filename), dtype= np.uint8)
    region_data= img_json_data["regions"]
    objects_list= []

    for region in region_data:
        objects, coordinates= getRegionProperties(region)
        if coordinates is not None:
            polygons.append(coordinates)
            polygon_coordinates[objects]= coordinates
            objects_list.append(objects)

    # Masking the images
    maskImage(filename, polygons, objects_list)
    return (filename, polygon_coordinates)

def main():

    root_path= 'path'

    json_file= open(root_path+'filename.json')
    json_data= json.load(json_file)

    # for multiprocessing
    pool = mp.Pool(mp.cpu_count()-1)

    def resultCallback(item):
        return

    for data in json_data:
        pool.apply_async(parallelizePlotting, args=(data, json_data), callback= resultCallback)

    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

very userful,thanks a lot.

[manvirvirk]

i m facing the same issue. My predicted image is a gray box. Can anyone help

[Cipherpy]

@BriFuture I will tell you the simple steps to create the masks. Assuming you have 2 classes:

1. Open the JSON file in python and process image by image.
2. Read the corresponding image from the disk.
3. Create a mask having the same size as the image using np.zeros()
4. I have not used labelme, but I have used VGG Annotation tool. Here you need to read the coordinates one by one for the corresponding image and then use cv2.drawContours() to fill that polygon with (255, 255, 255) on the mask. Do this for all the polygons.
5. Save the masks separately in a masks folder using cv2.imwrite().
6. Do this for all the images.

can you share me the code for this. I have multiple classes in a single image

[sainatarajan]

@Cipherpy Take a look here

[Cipherpy]

@Cipherpy Take a look here

Thank you

[kimile599]

@sainatarajan do you still remember where the data came from in the function of "def parallelizePlotting(data, region_mappings, json_data):"