Using ImageDataGenerator for images with more than 3 channels

[RaananHadar]

Currently, the ImageDataGenerator.fit() and ImageDataGenerator.flow() implementations require assert that the number of channels is either 1 or 3. Some datasets use images with a different number of channels while not being video. For example RGBD, RGB+NIR imagery (common in satellite imagery) or even 8 band multispectral imagery. A current example is Spacenet challenge dataset.

This was tested with the current theano and keras, current exception is at the implementation line 439.

[fchollet]

I've allowed 4-channel images. Please update Keras.

[RaananHadar]

@fchollet Thank you for the swift response. However, I suggest you consider a generic solution.

To illustrate the need, Spacenet is a current dataset using 8 bands. It uses the relatively old (yet common) Worldview 2 satellite. The more current Worldview 3 satellite has 16 channels. These are not niche' satellites, they are the mostly used commercial satelites in the world... Someone using Hyperspectral imagery with Keras will want to use images with hundreds of channels... You get my point.

[grantbey]

This went dead, but I'd like to co-sign it. I'm currently working with Worldview 3 data and I'd love to be able to use ImageDataGenerator.

[sqh4587]

@fchollet, hi, fchollet, how to set the parameter "color_mode" in API "flow_from_directory(directory)" in "ImageDataGenerator" class? "color_mode": one of "grayscale", "rbg". Default: "rgb". Whether the images will be converted to have 1 or 3 color channels. This is the comment from https://keras.io/preprocessing/image/

[Schlump]

Would like to consent my interest in that issue. If we look at remote sensing data nowadays it's common to have at least multi spectral data. And as finally such data is freely available (Sentinel-2 data) imho the general usage of such data will increase.

[sqh4587]

@fchollet @Schlump @grantbey @RaananHadar @bmabey dear all, since some of you had successfully test the 4 channel image with ImageDataGenerator, can you teach me how to set the flow_from_directory's parameter color_mode? the default is only support 'grayscale' and 'rgb'. If I set 'rgb', then when I set the input_shape= (200,200,4) to the first CNN layer, the error is shows as: ValueError: Error when checking input: expected CONV_1_input to have shape (None, 200, 200, 4) but got array with shape (60, 200, 200, 3)

my program is as following: `#here is set the ImageDataGenerator for train_datagen train_datagen = ImageDataGenerator( rescale=1./255, rotation_range=40, width_shift_range=0.2, height_shift_range=0.2, horizontal_flip=True, vertical_flip=True, fill_mode='nearest' ) train_generator = train_datagen.flow_from_directory( directory= train_dir, # this is the target directory target_size=(200, 200), # all images will be resized to 200x200 classes= potato_class, batch_size=60, color_mode= 'rgb', class_mode='sparse')

here is the model first CNN layer setting

model = Sequential() # CNN构建 model.add(Convolution2D( input_shape=(200, 200, 4),

input_shape=(1, Width, Height),

filters=16,
kernel_size=3,
strides=1,
padding='same',
data_format='channels_last',
name='CONV_1'

)) `

the error shows as : ValueError: Error when checking input: expected CONV_1_input to have shape (None, 200, 200, 4) but got array with shape (60, 200, 200, 3)

[RaananHadar]

@sqh4587 Scanning https://github.com/fchollet/keras/blob/master/keras/preprocessing/image.py I see multiple lines that throw warnings and raise value errors if the number of channels is not 3. Its a simple if clause that checks if the number of channels is 3 or not, otherwise it throws an error.

My quick scan shows lines: 257 and 277 may cause errors and 676,849 should throw warnings. I suggest you consider looking there.

Honestly I think @fchollet should treat this issue as one that needs fixing. not all images have 1,3 or 4 channels :) I know that you can probably cheat your way using a 3d convolution treating this as temporal data but thats probably not the proper way to solve this.

[sqh4587]

@RaananHadar Hi, RaannaHadar, thank you for the reply, Yesterday, I tried to use tensorflow to train the model for 4-channel image, but the loss and acc in training and validation is not as well as by keras with the similar model, now, I'm considering to cut one of RGB channel which is not very obvious in shape or texture and replace it by Depth channel. It's not as good as using keras to train 4-channel image, but as least it should work. @fchollet can you teach us how you train the 4 channels image? or leave us a link for study, thank you :)

[Schlump]

@sqh4587: As RaananHadar already stated if you exclude the lines he mentioned Keras should work with 4,5,6... channel images.. at least i did not notice any errors.

[marcelsan]

I'd like to express my interest in this issue too. There are many works related too PathTracing denoising which use more than 4 channels neural networks.

[gledsonmelotti]

@Schlump Line 257 is a comment. Do not interfere in anything. Which lines do I really need to comment on to work with more than 3 channels?

[gledsonmelotti]

@marcelsan Were you able to work with more than 4 channels?

[jfprieur]

Just would like to add my support to be able to use n-layers. I make rasters from airborne lidar data so I have an arbitrary number of layers I produce from the data eg. can be height and intensity (2 layers), can be 3-channel intensity from multi-spectral lidar (3 layers), height plus three channel intensity (4 layers) etc. Having a parameter where we can set the number of bands (instea dof the current 1 or 3) would be immensely helpful.

There are other ways of feeding the data manually that works but flow_from_directory is so elegant and useful it is a shame not to use it.

[raghu1121]

@fchollet @RaananHadar I am also in need of 4 channels . I have Keras 2.2.0 but i don't see any code changes related to the 4 channels. In my case i have HDR images, which have RGBE channels. It would be great, if could let me know where to make code changes.

Thanks

[wisdomikezogwo]

@jfprieur Please, what ways do you use to feed your n-channel data manually, I have a n-channel data from n-different frequency bands and would like to use CNN to represent its spatial information. @fchollet can this: https://github.com/minhnhat93/tf_object_detection_multi_channels solve my problem using TF as backbone

[LTBen]

Hey, I would love to see this feature as well. N-Channels for example as multi page Tiff file would be awesome.

[jwindhager]

Working on multichannel images as well, please consider implementing a generic solution.

[mabdelhack]

I have just implemented a library for hyperspectral image augmentation here