How to Create a Multi-inputs Convolutional Neural Network Model for Images Classification?

[nadaalay]

I am beginner in deep learning and I hope if you help me to solve my issue.

I want to create a CNN model that takes two inputs of images and produce one output which is the class of the two images. The model takes one image from dataset type1 and one image from dataset type2. I have two datasets: type1 and type2, and each dataset contains the same number of classes, but the number of images in each class in the dataset type1 is higher than the number of images in each class in the dataset type2. The following is the structure of the datasets.

The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to one class (ClassA or ClassB or------).

Type1 dataset
|Train
              |ClassA
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassB
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassC
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassD
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

Type2 dataset
|Train
              |ClassA
                             |image1
                             |image2
                            -----
              |ClassB
                             |image1
                             |image2
                            -----
              |ClassC
                             |image1
                             |image2
                            -----
              |ClassD
                             |image1
                             |image2
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

So, I want to create a model that inputs two images (from type 1 and 2), as long as they are from the same class. Also, I want each image from type1 be paired with every images from type2 from the same class. How can I do this???

The code:

in1 = Input(...)  
 x = Conv2D(...)(in1)
--------
--------
 out1 = Dense(...)(x)  

 in2 = Input(...)  
x = Conv2D(...)(in2)
--------
--------
out2 = Dense(...)(x)  

concatenated_layer = concatenate([out1, out2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(concatenated_layer)
modal= Model(inputs=[in1, in2], outputs=[output_layer])

input_imgen = ImageDataGenerator(rescale = 1./255, 
                                   shear_range = 0.2, 
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator()

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)       

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=1,
                                          img_height=224,
                                          img_width=224)  

[csitaula]

@nadaalay Did you solve this problem yet? I am also trying to do so. Look forward to hearing from you. Thanks

[nadaalay]

@nadaalay Did you solve this problem yet? I am also trying to do so. Look forward to hearing from you. Thanks

Yes, I solved the problem. I arranged the images in the dataset (Type1 and Type2) according to the way that I wanted the images to enter the model and the model receives the images from the dataset without shuffling.

[csitaula]

Can you share your code? It would be wonderful to play with it. I am planning to design such a network for multiple patches extracted from the images. Thanks in advance.

---

Best Regards, Chiranjibi Sitaula Postgraduate Research Fellow School of IT Burwood Campus Deakin University,Melbourne, Australia ORCID:orcid.org/0000-0002-4564-2985 CYA E-RYT 550 Certified Yoga Instructor from Nepal.

On Thu, Jun 13, 2019 at 6:04 PM Nada notifications@github.com wrote:

@nadaalay https://github.com/nadaalay Did you solve this problem yet? I am also trying to do so. Look forward to hearing from you. Thanks

Yes, I solved the problem. I arranged the images in the dataset (Type1 and Type2) according to the way that I wanted the images to enter the model and the model receives the images from the dataset without shuffling.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/keras-team/keras/issues/11607?email_source=notifications&email_token=AIRLZ2VWNWZ4XQUKMVPQBHTP2H5QPA5CNFSM4GCS25W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXS33FQ#issuecomment-501595542, or mute the thread https://github.com/notifications/unsubscribe-auth/AIRLZ2SBKRVQDISVNN5PHD3P2H5QPANCNFSM4GCS25WQ .

[cj00719]

@nadaalay Please I am having a similar challenge for multiple inputs and single output.

Can you kindly share your code?

[nadaalay]

The following codes would be helpful, try them and do not hesitate if you have any questions.

This the code of image generator for three inputs

train_size = 5300

test_size = 1802

input_imgen_1 = ImageDataGenerator(

                                  rotation_range=10,

                                  shear_range=0.2,

                                  zoom_range=0.1,

                                  width_shift_range=0.1,

                                  height_shift_range=0.1

                                 )

input_imgen_2 = ImageDataGenerator(

                                  rotation_range=10,

                                  shear_range=0.2,

                                  zoom_range=0.2,

                                  width_shift_range=0.1,

                                  height_shift_range=0.1,

                                  horizontal_flip=True

                                 )

input_imgen_3 = ImageDataGenerator(

                                  rotation_range=10,

                                  shear_range=0.2,

                                  zoom_range=0.1,

                                  width_shift_range=0.1,

                                  height_shift_range=0.1,

                                  horizontal_flip = True

                                  )

test_imgen = ImageDataGenerator()

bs = 64

# Create generator for multiple inputs

def generate_generator_multiple(1_generator, 2_generator,  3_generator, dir1, dir2, dir3, batch_size, img_height,img_width):

    gen_1 = 1_generator.flow_from_directory(dir1,

                                          target_size = (img_height,img_width),

                                          class_mode = 'categorical',

                                          batch_size = batch_size,

                                          shuffle=False, 

                                          seed=7)

    gen_1.reset()

    gen_2 = 2_generator.flow_from_directory(dir2,

                                          target_size = (img_height,img_width),

                                          class_mode = 'categorical',

                                          batch_size = batch_size,

                                          shuffle=False, 

                                          seed=7)

    gen_2.reset()

    gen_3 = 3_generator.flow_from_directory(dir3,

                                          target_size = (img_height,img_width),

                                          class_mode = 'categorical',

                                          batch_size = batch_size,

                                          shuffle=False, 

                                          seed=7)

    gen_3.reset()

    while True:

            1_ = gen_1.next()    

            2_ = gen_2.next()

            3_ = gen_3.next()

            yield [1_[0], 2_[0], 3_[0]], 2_[1]  #Yield images and their label

multi_train_generator=generate_generator_multiple(1_generator=input_imgen_1,

                                           2_generator=input_imgen_2,

                                           3_generator=input_imgen_3,

                                           dir1=train_1_data2,

                                           dir2=train_2_data,

                                           dir3=train_3_data2,

                                           batch_size=bs,

                                           img_height=224,

                                           img_width=224)       

multi_valid_generator=generate_generator_multiple(1_generator=test_imgen,

                                           2_generator=test_imgen,

                                           3_generator=test_imgen,

                                           dir1=valid_1_data2,

                                           dir2=valid_2_data,

                                           dir3=valid_3_data2,

                                           batch_size=bs,

                                           img_height=224,

                                           img_width=224) 

multi_test_generator=generate_generator_multiple(1_generator=test_imgen,

                                          2_generator=test_imgen,

                                          3_generator=test_imgen,

                                          dir1=test_1_data2,

                                          dir2=test_2_data,

                                          dir3=test_3_data2,

                                          batch_size=1,

                                          img_height=224,

                                          img_width=224)

This the code of model of three inputs :

# Merge the fully connected layers of the two models
concatenated_layer1 = concatenate([fc12, fc22, fc32])  
dropout = Dropout(0.4, name='dropout1')(concatenated_layer1)

output = Dense(no_classes, activation='softmax', name='prediction4')(dropout)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2, input_layer3], outputs=[output])

[cj00719]

@nadaalay Thank you very much for the shared code. It very much appreciated. I will implement and revert back if there is an issue. Thank you

[cj00719]

@nadaalay

Please "fitting the model" possess an issue.

Please kindly assist. My email is: kmanifestation@gmail.com

Thank you

[cj00719]

@nadaalay

Please, can you help provide the sequence for the model.fit_generator ?

Thank you

[Ijustakid]

@nadaalay hi, your sharing is very helpful to me, thanks

Can you share how to write fit_generator function for multiple inputs?

Thanks

[Luca1236a]

Hi, I'm trying with multi-modal input. I have 2 inputs: image and gender. How can I use ImageGenerator in this case?

[KaterinaBao]

@nadaalay

Please, can you help provide the sequence for the model.fit_generator ?

Thank you

Hi, when I create this model, there is an valuerError about 'Data cardinality is ambiguous, please provide data which shared the same first dimension. I think it may be the problem of 'model.fit...'. Have you already solve the problem?

Thanks!

[samiqazi]

@nadaalay Can you share the complete code, pls. Thanks.

[aneeka657]

@nadaalay

Please "fitting the model" possess an issue.

Please kindly assist. My email is: kmanifestation@gmail.com

Thank you

Hello , did you get model.fit for this code? please share if you have

[kaziomarfaruk]

@nadaalay can you please share the code dipsgalaxy01@gmail.com ?

[Aditijain114]

how to get labels/classes from these generator multi_train_generator.labels()