ImageDataGenerator With 2 channel data?

[varoudis]

Any plans for allowing arbitrary number of channels in the processing utils? Thanks

[madeofwin]

Have you investigated if and where it fails with 2 channel data? If I remember correctly, the apply_transform method is applying the transformation to each channel separately. In fact, I have used it with data of different channels (2 also), but only by calling the transformation methods (shifting, rotating, etc.) directly, not via ImageDataGenerator.

[varoudis]

To be honest I haven't as I though 99% of training is done with 1 or 3 channels from images. Reading the docs was also stating this.

Ill come back with results on this.

[oeway]

hi @varoudis , you are right, currently ImageDataGenerator only support 1 and 3 channels. But I am working on extending it in many aspects, check #3338 , one goal is to extend it to support arbitrary number of channels, I now have a working version, you are welcome to try and provide feedback.

[varoudis]

that sounds great!

[gledsonmelotti]

@oeway In the keras command, you have to define the image type, that is, color_mode = 'rgb' or 'grayscale' when using the command flow_from_directory with model.fit_generator. So how do I change color_mode in flow_from_directory to use with 2, 4, 5 and 6 channels?

I thank you for your attention, Gledson

[oeway]

@gledsonmelotti You can try with my extension: https://github.com/keras-team/keras/issues/3338 . For images with 2, 4, 5 and 6 channels or more, you need to define your own image_reader.

This is currently we used by default:

def pil_image_reader(filepath, target_mode=None, target_size=None, dim_ordering=K.image_dim_ordering(), **kwargs):
    img = load_img(filepath, target_mode=target_mode, target_size=target_size)
    return img_to_array(img, dim_ordering=dim_ordering)

You can define your own reader, for example, you can load your image in tif format which can contain multiple frames other than 1 or 3.

def customized_image_reader(filepath, target_mode=None, target_size=None, dim_ordering=K.image_dim_ordering(), **kwargs):
    imgTiff = PIL.Image(filepath, mode="r")
    imgs = []
    for i in range(6):
        imgTiff.seek(i)
        img = np.array(imgTiff, dtype="float32")
        imgs.append(img)
    imgArr = np.stack(imgs)
    return imgArr

Then you can use it like this:

datagenX.flow_from_directory(image_folder, image_reader=customized_image_reader, ...)

Please also notice that there maybe compatible issue with latest Keras, you may need to adjust the code yourself.

[thepate94227]

@oeway the flow.from.directory function doesn't have a parameter called image_reader... I have the same problem. I have a stack of image labels, in this case to grayscale images. So the stack has the shape (300,200,2). I don't know how to train my model, when i can't use flow_from_directory function...

[oeway]

@thepate94227 That's because my extension is not in the official implementation, perhaps you want to look at this: https://github.com/bernardohenz/ExtendableImageDatagen a further extension made by @bernardohenz.

[thepate94227]

@oeway I tried to do it your way, but i get an error in this line

imgTiff = PIL.Image(filepath, mode="r")
TypeError: 'module' object is not callable

Do you know why? It seems, that PIL.Image is not a function... Usually i use flow_from_directory and my Filepath is like: /Images/ And there are 2 Folders: Input and Label. In each Folder there is another folder called "0". So basically i have 2 folders /Images/Input/0/ and /Images/Label/0/. This two folders contain my images respectively labels. Which Filepath do i have to use for my custom_filereader? I tried /Images/ like i use in Keras, and i tried /Images/Label/ and /Images/Label/0/, because i only need the custom_filereader for my labels. But i always get the TypeError.

This is my code so far:

NAME = "DeepLab-{}".format(int(time.time()))
deeplab_model = Deeplabv3(input_shape=(300,200,3), classes=2)
tensorboard = TensorBoard(log_dir="/logs/{}".format(NAME))
deeplab_model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=['accuracy'])

def customized_image_reader(filepath, target_mode=None, target_size=None, dim_ordering=K.image_dim_ordering(), **kwargs):
    imgTiff = PIL.Image.open(filepath, mode="r")
    imgs = []
    for i in range(6):
        imgTiff.seek(i)
        img = np.array(imgTiff, dtype="float32")
        imgs.append(img)
    imgArr = np.stack(imgs)
    return imgArr

image_reader = customized_image_reader(filepath='/Keras3/Label/',
                                                  target_size=(300,200,2),
                                                  target_mode=None)

# we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
                     featurewise_std_normalization=True,
                     rotation_range=90,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2)

image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1
#image_datagen.fit(images, augment=True, seed=seed)
#mask_datagen.fit(masks, augment=True, seed=seed)

image_generator = image_datagen.flow_from_directory(
    'Keras3/Input/',
    target_size=(300,200),
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    class_mode=None,
    seed=seed,
    image_reader=image_reader)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

deeplab_model.fit_generator(train_generator, steps_per_epoch= np.uint32(2935 / 32), epochs=10, callbacks=[tensorboard])

deeplab_model.save_weights('deeplab_6.h5')
deeplab_model.save('deeplab-6')

session.close()

[oeway]

You should do from PIL import Image and img = Image.open(filepath).

[thepate94227]

You should do from PIL import Image and img = Image.open(filepath).

Dann i get an error: IsADirectoryError: [Errno 21] Is a directory: '/Keras3/Label/0/'

[gledsonmelotti]

@gledsonmelotti You can try with my extension: #3338 . For images with 2, 4, 5 and 6 channels or more, you need to define your own image_reader.

This is currently we used by default:

def pil_image_reader(filepath, target_mode=None, target_size=None, dim_ordering=K.image_dim_ordering(), **kwargs):
    img = load_img(filepath, target_mode=target_mode, target_size=target_size)
    return img_to_array(img, dim_ordering=dim_ordering)

You can define your own reader, for example, you can load your image in tif format which can contain multiple frames other than 1 or 3.

def customized_image_reader(filepath, target_mode=None, target_size=None, dim_ordering=K.image_dim_ordering(), **kwargs):
    imgTiff = PIL.Image(filepath, mode="r")
    imgs = []
    for i in range(6):
        imgTiff.seek(i)
        img = np.array(imgTiff, dtype="float32")
        imgs.append(img)
    imgArr = np.stack(imgs)
    return imgArr

Then you can use it like this:

datagenX.flow_from_directory(image_folder, image_reader=customized_image_reader, ...)

Please also notice that there maybe compatible issue with latest Keras, you may need to adjust the code yourself.

@oeway thanky very much. It is perfect.

[gledsonmelotti]

You can find a good idea in https://github.com/keras-team/keras/issues/10499 and https://stackoverflow.com/questions/49404993/keras-how-to-use-fit-generator-with-multiple-inputs. They use two ImageDataGenerator.

[thepate94227]

You can find a good idea in #10499 and https://stackoverflow.com/questions/49404993/keras-how-to-use-fit-generator-with-multiple-inputs. They use two ImageDataGenerator.

Thank you, but my problem is, that i have semantic segmentation, so i have one input folder with many images and i have labels for each class. For example: i have an image with a human, a cat and a background. For that image i have a ground truth with two labels: one for the human, one for the cat. I ignore the background. When i use Keras flow_from_directory, i can only use images with 1, 3 or 4 channels, but for my task i have the two labels stacked together, so it has 2 channels...

[gledsonmelotti]

You can find a good idea in #10499 and https://stackoverflow.com/questions/49404993/keras-how-to-use-fit-generator-with-multiple-inputs. They use two ImageDataGenerator.

Thank you, but my problem is, that i have semantic segmentation, so i have one input folder with many images and i have labels for each class. For example: i have an image with a human, a cat and a background. For that image i have a ground truth with two labels: one for the human, one for the cat. I ignore the background. When i use Keras flow_from_directory, i can only use images with 1, 3 or 4 channels, but for my task i have the two labels stacked together, so it has 2 channels...

@thepate94227 You can split your 2 channels images into one channel images. Create a folder for the first channel and a folder for the second channel. You can save each channel as a new png file. Then you can use the following commands:

train_datagen = ImageDataGenerator(rescale=1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):

genX1 = generator.flow_from_directory(dir1,
                                      target_size = (img_height,img_width),
                                      color_mode='grayscale',
                                      class_mode = 'categorical',
                                      batch_size = batch_size)

genX2 = generator.flow_from_directory(dir2,
                                      target_size = (img_height,img_width),
                                      color_mode='grayscale',
                                      class_mode = 'categorical',
                                      batch_size = batch_size)
while True:
        X1i = genX1.next()
        X2i = genX2.next()
        imgFirst = X1i[0]
        imgSecond = X2i[0]
        imgi=np.zeros((len(imgFirst), img_width, img_height, img_channels), dtype=np.float32)
        for n in range(0,len(imgFirst)):
            img1 = imgFirst[n]
            img2 = imgSecond[n]
            imgi[n] = np.dstack((img1,img2))
        yield imgi, X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=train_datagen, dir1=train_dir_First_Channel, dir2=train_dir_Second_Channel, batch_size=batch_size, img_height=img_height, img_width=img_height)

Note that I did this with the explanations of the following sites:

10499 and https://stackoverflow.com/questions/49404993/keras-how-to-use-fit-generator-with-multiple-inputs

[thepate94227]

Thank you for you answer. I think i understand and will try it! And my Generator for my Input Images can be like in the Keras Tutorial? Like this?:

#we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
                     featurewise_std_normalization=True,
                     rotation_range=90,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2)

image_datagen = ImageDataGenerator(**data_gen_args)

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'Keras3/Input/',
    target_size=(300,200),
    class_mode=None,
    seed=seed)

And then i combine them like in the Tutorial?

# combine generators into one which yields image and masks
train_generator = zip(image_generator, custom_output_generator)

With custom_output_generator like your code. Or do i have to edit my Generator for the Input Images, too?

[gledsonmelotti]

Thank you for you answer. I think i understand and will try it! And my Generator for my Input Images can be like in the Keras Tutorial? Like this?:

#we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
                     featurewise_std_normalization=True,
                     rotation_range=90,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2)

image_datagen = ImageDataGenerator(**data_gen_args)

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'Keras3/Input/',
    target_size=(300,200),
    class_mode=None,
    seed=seed)

And then i combine them like in the Tutorial?

# combine generators into one which yields image and masks
train_generator = zip(image_generator, custom_output_generator)

With custom_output_generator like your code. Or do i have to edit my Generator for the Input Images, too?

Hello @thepate94227

Suppose you have 2 main files: Channel 1 and Channel 2. Channel 1 will have: 1) Training file 2) Test file 3) Validation file

Channel 2 will have: 1) Training file 2) Test file 3) Validation file

dir1 = training subfile or validation subfile of Channel1. (path directory of teh subfile) dir2 = training subfile or validation subfile of Channel2. (path directory of the subfile)

Using the commands below, Keras alone will recognize the subfiles and keras will create the classes that exist in each subfile without you having to do anything.

batch_size=64 num_epochs=30 DROPOUT = 0.5 num_classes = 3

img_width, img_height, img_channels = 200, 200, 2

model_input = Input(shape = (img_width, img_height, img_channels)) x= Convolution2D...

your model

model_output = Dense(num_classes, activation='softmax')(z) model = Model(model_input, model_output) model.summary()

sgd = SGD(lr=0.001, decay=1e-6, momentum=0.9, nesterov=True) model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

genX1 = generator.flow_from_directory(dir1, target_size = (img_height,img_width), color_mode='grayscale', class_mode = 'categorical', batch_size = batch_size)

genX2 = generator.flow_from_directory(dir2, target_size = (img_height,img_width), color_mode='grayscale', class_mode = 'categorical', batch_size = batch_size) while True: X1i = genX1.next() X2i = genX2.next() imgFirst = X1i[0] imgSecond = X2i[0] imgi=np.zeros((len(imgFirst), img_width, img_height, img_channels), dtype=np.float32) for n in range(0,len(imgFirst)): img1 = imgFirst[n] img2 = imgSecond[n] imgi[n] = np.dstack((img1,img2)) yield imgi, X2i[1] #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=train_datagen, dir1=train_dir_First_Channel, dir2=train_dir_Second_Channel, batch_size=batch_size, img_height=img_height, img_width=img_height)

validationtgenerator=generate_generator_multiple(generator=train_datagen, dir1=train_dir_First_Channel, dir2=train_dir_Second_Channel, batch_size=batch_size, img_height=img_height, img_width=img_height)

Results_Train=model.fit_generator(inputgenerator, steps_per_epoch=nb_train_samples // batch_size, epochs = num_epochs, validation_data = validationtgenerator, validation_steps = nb_validation_samples//batch_size, callbacks=[History, checkpointer, csv_logger], shuffle=True, verbose=1)

[thepate94227]

@gledsonmelotti Thank you very much. I looked through your code, but i don't see how i can use it for the output images. I understand that if i have two input, i can do it this way. But my problem is the other way around. I explain my task: I have input like this: You can see two ropes and the background. I want my NN to show me the three parts. Rope Blue, Rope Red, Green Background. But without the color information. Therfore i convert my image to grayscale:

I use a Semantic Segmentation NN called DeepLab: https://github.com/bonlime/keras-deeplab-v3-plus My goal is that this already trained DeepLab can give me an output, which looks like this:

DeepLab want the labels to be stacked. For example if i have 20 classes like class rope, class human, class cat etc. then the ground thruth is a stacked image with shape (300, 200, 20).

I have two classes, rope blue and rope red. The background is not important. Therfore my ground thruth are images with (300,200,2). But with Keras flow_from_directory it can't read the 2 channels.

Your solution is good, if i have two input images for the same label. But i only want one input image and one label, but the label is a stacked image of all the labels. If i have only one class, my label would have the shape (300,200). I i have 27 classes, my label would have the shape (300,200,27). I don't know why DeepLab wants it this way, but it is how it is.

You can see my code above: https://github.com/keras-team/keras/issues/3416#issuecomment-467876435

I tried it like oeway said, but i got an error...

I hope you understand, what i mean.

[gledsonmelotti]

Hello @thepate94227 Now I understand the your problem. Unfortunately I do not know how to solve or help you. I am sorry.

[thepate94227]

Hello @thepate94227 Now I understand the your problem. Unfortunately I do not know how to solve or help you. I am sorry.

No problem! Thank you for your help and answers :)

[cyrilvincent]

To be compatible with Tensorflow 2.1 and the new fit method (fit_generator is deprected) I propose this implementation :

` class TwoImageGenerator(keras.utils.Sequence):

def __init__(self, generator1, generator2):
    self.generator1 = generator1
    self.generator2 = generator2

def __getitem__(self, i):
    X1i = self.generator1[i]
    X2i = self.generator2[i]
    imgFirst = X1i[0]
    imgSecond = X2i[0]
    imgi = np.zeros((len(imgFirst), 64, 64, 2), dtype=np.float32)
    for n in range(0, len(imgFirst)):
        img1 = imgFirst[n]
        img2 = imgSecond[n]
        imgi[n] = np.dstack((img1, img2))
    return imgi, X2i[1]

def __len__(self):
    return len(self.generator1)

`

using with

model.fit( TwoImageGenerator(trainGenerator1, trainGenerator2), epochs=30, validation_data=TwoImageGenerator(validationGenerator1, validationGenerator2),