npz format for gen_images_and_labels

[mvilar22]

Hi there, I have about 15k images that are already labelled, in order to use this images with gen_images_and_labels.py I need a npz file for each image. I have managed to create them like this:

i = 0
for filename in os.listdir(src):
    if filename[-3:] == "png":
        npz = {}
        #open the image, turn it into a numpy array 
        img = cv2.imread(src+filename.split(".")[0]+'.jpg')
        img = np.asarray(img)
        npz['image'] = img
        #open the label, turn it into a numpy array
        label = cv2.imread(src+filename)
        label = np.asarray(label)

        npz['label'] = label
        npz['classes'] = ['vegetation' 'background']
        np.savez_compressed(src+filename.split(".")[0]+'.npz', npz, **npz)
        print(i)
        i+=1

However, when I run the script I check the overlay images generated and they are "blue" with only one class present.

The labels I am feeding are png like this:

I don't know if this is correct, the script seems to one-hot encode the labels so I don't think I should feed the script labels already encoded. What am I missing?

Thanks!

[ebgoldstein]

Hi @mvilar22 - can you provide a few example image-label pairs for us to experiment with?

[dbuscombe-usgs]

Hi @mvilar22 , any reason why you can't use "make_nd_daraset.py" for this task? It will ask for your greyscale labels and images, and make npz files for gym. You need greyscale labels. If you need to make your own npz files, the arrays must be called 'arr_0' and 'arr_1'

[mvilar22]

Hi @ebgoldstein, here are some examples. The images are taken from time lapse videos of plants growing. There are only 2 classes, background and vegetation.

fewExamples_PepperTimeLapse.zip

@dbuscombe-usgs haven't really thought about that, so I can skin this "step" and use make_nd_dataset.py directly with the one-hot encoded labels and the images?

I was just trying to recreate the process as if I had doodled the images but now it seems a bit redundant

[mvilar22]

Hi there, so I did as @dbuscombe-usgs suggested and I ran into some issues.

I one hot encoded the labels and ran make_nd_dataset.py but an error popped up:

The script did resize the images since I have a resized_images folder with all the images but the resized_labels folder is empty and the script says there are 0 labels, as you can see in the photo.

Maybe there is a problem in the one hot encoding process, I use this code to create the jpg labels:

for filename in tqdm.tqdm(os.listdir(src), desc="One-hot encoding labels"):
    #Load image and turn it to RGB  
    img = cv2.imread(os.path.join(src, filename))
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    #Find the image dimensions
    h, w, c = img.shape

    #Define the colors that represent classes
    blue = (51,102,204)
    red = (220, 57, 18)

    #Create masks for each class
    mask_1 = np.all(img==blue, axis=-1)
    mask_2 = np.all(img==red, axis=-1)

    #Create a class map assigning indexes according to the class masks 
    indices = np.zeros((h,w), dtype=np.uint8)
    indices[mask_1] = 1
    indices[mask_2] = 0
    #one-hot array 
    nx, ny = indices.shape
    lstack = np.zeros((nx, ny, 2))
    #Assign values
    lstack[:,:,:2] = (np.arange(2) == indices[...,None]-1).astype(int)
    #Obtain the labels
    l = np.argmax(lstack,-1).astype('uint8')

    cv2.imwrite(output_path + filename, l)

The final part of the code comes from gen_images_and_labels.py I did test the code with an image and label I had already labelled using dashdoodler and it produced the same jpg, I checked comparing each individual channel of both images with opencv.

Any insight into what am I surely doing wrong?

Thanks!

[dbuscombe-usgs]

Can you provide a small sample of images and corresponding labels? It's not clear to me why you need to use opencv commands to make jpegs

[dbuscombe-usgs]

If the program can't find the labels, it is probably because you have either named them in such a way that the can't be paired with imagery using a natural sort, or the images are in the wrong folder structure or something

[ebgoldstein]

Hi both -

I have some time today where i can take a look at this issue using the sample images that @mvilar22 provided in a zip file (upthread)

sorry for my delay

[ebgoldstein]

@mvilar22 - I am looking at the images in the zip file.

make_nd_dataset.py uses image-label pairs where the image is an RGB jpg and the label is just a greyscale jpg. (The label file is not one-hot encoded, it is just 'label encoded')...

In theory the png file you provided in the zip file will work, but they need to be adjusted in a few ways:

1. make them jpgs (not png)
2. you will need to convert all labels to greyscale where the value of each pixel corresponds to the class. so for 2 classes, the pixels should just be 0 or 1.. If you open up a label with an image viewing software, the label will look all black..
3. add a _label.jpg to the file name, so the image is 7470_PepperTimeLapse.jpg and the label is 7470_PepperTimeLapse_label.jpg.
4. make sure the data is in the correct directory structure (see here : https://github.com/Doodleverse/segmentation_gym/wiki/03_Directory-Structure-and-Tests)

After you have done these steps, you should be able to run make_nd_dataset.py. the output of that script will be a series of npz's that can be used for train_model.py

Also, Q for you - were these image-label pairs made using Doodler ? if so, you can use utils/gen_images_and_labels.py to create the images and labels for you automatically, then just drag/drop them into the Gym directory and run make_nd_dataset.py

Does this make sense?

[mvilar22]

Hi both, thanks for the answers,

The zip I provided earlier had some images and pngs with just the 2 colors, my last attempt was with the grayscale jpgs, that I made using the last code snippet.

I guess my previous posts were a bit confusing, I will try to explain it better. Initially I was using Doodler to label this images, however, I was too slow for the volume of images I am working with. As a result, I was forced to automate the labelling process using more traditional approaches. That's how I ended up with 15k labels. I emulated (or at least that was what I tried to do) the labels that Doodler produces, this red and blue pngs. However, I realized by looking at gen_images_and_labels.py that I needed npz for it to work and that was my initial doubt, because I was unable to succesfully make the npz as is its done by Doodler. Later, as @dbuscombe-usgs suggested, I transformed this red and blue pngs into grayscale jpg labels (with the code I provided last message) and feed them to make_nd_dataset.py. While the script did accept the images, it did not seem to recognize the labels.

Hopefully this makes more sense, now for the questions:

I use opencv because I am simply more familiar with it. I think is a really useful library, but if needed I can try to use another one, I think Doodler uses scikit-image.

The directory structure I follow is the one in the wiki, I have a crops_data folder with several subfolders: fromDoodler, npzForModel, toPredict. Inside the fromDoodler folder I have 2 extra folders, images and labels. In this last two folders I have the jpgs for the images and the jpgs for the grayscale labels.

I will rename the labels and try again with make_nd_dataset.py, and perhaps give it a try again with gen_images_and_labels.py and see if this time I manage to get it to work, since last time it didn't seem to work

I have attached a zip with some image-labels pairs, this time the ones I use with make_nd_dataset.py toy-labels.zip

Thanks a lot for the answers and sorry to be such a bother :(

[ebgoldstein]

no bother at all!

yes - try to rename the files, make them jpegs.. then try make_nd_dataset.py

let me know how it works!

[mvilar22]

As it turns out, the labels were indeed pngs, I could have sworn they were jpgs but I stand corrected, now that they are jpgs make_nd_dataset.py does work!

I am running into a little problem when runningtrain_model.py with the ptxas binaty, the path and a message saying there is no space left on the device, but that seems like a me problem so I am going to close the issue.

Thank you so much for the help!!

[ebgoldstein]

:+1: - keep us posted how you get on!

[mvilar22]

While the training completed without issue, it seems my labeling process was wrong. All the images feed to the net had their pixels belong to a single class, which is obviously wrong. make_nd_dataset.py does tells me the images belong to 1 classes, but if I recall correctly it says that even if there are more classes.

This is how I was making the grayscale jpg labels:

    #Load the image and convert it to RGB  
    img = cv2.imread(os.path.join(src, filename))
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    #Get the dimensions of the image
    h, w, c = img.shape

    #Define the colors representing the classes
    blue = (51,102,204)
    red = (220, 57, 18)

    #Create masks for each class
    mask_1 = np.all(img==blue, axis=-1)
    mask_2 = np.all(img==red, axis=-1)

    #Create a class map with the dimensions of the image, assigning indices based on the class masks
    indices = np.zeros((h,w), dtype=np.uint8)
    indices[mask_1] = 1
    indices[mask_2] = 0
    #Create label matrix with the dimensions of the class map
    nx, ny = indices.shape
    lstack = np.zeros((nx, ny, 2))
    #Assign values
    lstack[:,:,:2] = (np.arange(2) == indices[...,None]-1).astype(int)
    #Get the labels
    l = np.argmax(lstack,-1).astype('uint8')

    cv2.imwrite(output_path + filename.split('.')[0]+'.jpg', l)

The images that I am working with are png with only two colors, like this:

I have seen that the scripts have been updated recently and the there is a new make_dataset.py, but the changes don't seem to be related with my issue.

While debugging the code, up to the creation of indices it seems to work as intended, because if I apply indices as a mask I get only the blue pixels. That leads me to believe the problem resides in these lines:

    nx, ny = indices.shape 
    lstack = np.zeros((nx, ny, 2)) 
    lstack[:,:,:2] = (np.arange(2) == indices[...,None]-1).astype(int) 
    l = np.argmax(lstack,-1).astype('uint8')

As I understand it, lstack should be an int matrix with the same dimensions as indices. It should be filled with the result of np.arange(2)== indices[..,None]-1 that produces either True or False.

Since it is turning it into ints, the booleans turn to 0 or 1. indices[..,None]-1 should be -1 for the values that were 0 in indices and 0 for the values that were 1.

Then in the comparison, you get [0 0]for a value of -1 and [1 0] for the value of 0. Then l is created with the index for the maximum value of lstack. If I am not mistaken that is never going to be 1, because lstack is either [0 0] or [1 0] so the max is always in the position 0.

However, this code snippet comes from gen_images_and_labels.py that does work, so I don't know what I am missing.

In the mean time I will update to use the new scripts and try training again, but without the -1 when the values are assigned to lstack

[ebgoldstein]

@mvilar22 - I want to clarify, make_dataset.py does not take one-hot encoded labels, it takes grayscale labels (two dimensional jpgs, so M x N) . I think you need to adjust your code above to make sure it is not creating one-hot encoded labels...

see the reply above (https://github.com/Doodleverse/segmentation_gym/issues/130#issuecomment-1563073611) - when you open the labels, they should look all black but when you look at an array of the jpg, you should see different pixel values (o or 1 in your case)

[mvilar22]

Hi @ebgoldstein, thanks for the answer, I am sorry if I gave the impression I was one-hot encoding the labels or if I expressed myself incorrectly. The code I posted does say that i am one-hot encoding, but the result is a grayscale image.

Printing the array does not show it entirely (only zeroes) but this np.any(np.not_equal(img, 0)) tells me not all values are zero, so at some point I should have values of 1(and np.any(np.equal(img, 1)) does give me True). If I print the shape, I get (1080,1920) which in my case is (M x N).

I will edit the previous post to reflect that the code does not one-hot anything, sorry again

[ebgoldstein]

:+1: - can you post a few image-label pairs that you are giving to make_dataset? and also the config file for training the model?

Also - just to isolate the current issue you are having - is it that the trained model is not working in predict mode? do you see an error message?

[mvilar22]

The issue is that every pixel of every image belong to the same class. I haven't tried to predict on the test dataset because every picture provided in the training and validation dataset only had one class and the training was useless.

In the zip attached I have included some images, labels and the config file. samples.zip

I am training now with a modification to the code I posted before, but apparently using mirrored strategy eagerly has significant overhead and each epoch seems to be about 8h so I don't know if I have actually solved the issue.

Here are the same image-label pairs but with the "new" code, just in case. rectified-samples.zip

[ebgoldstein]

Hi @mvilar22

First, the labels in the samples.zip folder appear as all 0 to me... so something was wrong in the code to make those..

from PIL import Image
import numpy as np
from matplotlib import image
from matplotlib import pyplot

filename = "samples/labels/59_SunflowerTimeLapse.jpg"
with Image.open(filename) as im:
    width, height = im.size
    pixel_values = list(im.getdata())

print(np.unique(pixel_values))
pyplot.imshow(im)
pyplot.show()

However, in the rectified sample folder, i do see multiple values in the image. so the code you used for these images seems to work:

filename = "rectified-samples/labels/59_SunflowerTimeLapse.jpg"
with Image.open(filename) as im:
    width, height = im.size
    pixel_values = list(im.getdata())

pyplot.imshow(im)
pyplot.show()

Gym should work with these images in the rectified_samples folder (but note that these do not have _label in the name of the file, so you might want to add that.

At this point - if it's ok with you - i am going to close the issue : you have made images that should work with Gym.. If you run into a problem with training, let us know (and include the config, sample images).

let us know and good luck