RGB images for training

[matheuscass]

Hi! Is it possible to use RGB images (.jpg) to train the u-net? If possible, what part of the code do I need to modify? Thanks!

[Mithosk93]

Same problem, did you find a solution?

[ShawDa]

You can see here：https://github.com/ShawDa/unet-rgb

[teamo429]

i) in your model.py input_size (256,256,1) => (256,256,3) ii) trainGenerator function in your data.py image_color_mode ="grayscale"=>"rgb" iii) testGenerator function in your data.py
comment out "img = np.reshape(img,img.shape+(1,)) if (not flag_multi_class) else img"

[frankdeepl]

Hi @teamo429 ,

I tried to use your method to train model and predict the rpg image, however, all of predict image are black... do you know how to resolve it?

[alex337]

Hi @teamo429 ,

I tried to use your method to train model and predict the rpg image, however, all of predict image are black... do you know how to resolve it?

did you resolve it

[deaspo]

If anyone still has this issue with reading RGB images or getting errors please look for me. Went through the same and found a fix for ti

[sainatarajan]

If anyone still has this issue with reading RGB images or getting errors please look for me. Went through the same and found a fix for ti

Can you please tell how did you solve this issue. I am getting all white images as output.

[reachsak]

If anyone still has this issue with reading RGB images or getting errors please look for me. Went through the same and found a fix for ti

Hi , how did you resolve this issue

[deaspo]

I went through the following:

1. Assuming your data is 3-channel images, else you need to convert to 3-channel images. I had only gray images and converted first to 3-channel images
2. in the fn trainGenerator, when calling it, pass the parameter as image_color_mode='rgb' and also mask_color_mode='rgb' if the mask images are also 3-channels.
3. Before proceeding, check with the dataPrepare notebook that the trainGenerator actually generates correct augmented images by setting the path for save_to_dir. By correct I mean also 3-channels as the input used.
4. If everything is okay, then the input_size to the model pass as model = unet(input_size = (X,Y,3))

The next change will be on the testGenerator fn

1. Comment this line 89 --> #img = trans.resize(img,target_size), this line always resixe the input to (x,Y,1), a one channel shape. So avoid it when your input is 3-channel
2. When calling testGenerator fn, pass the following parameters as follows: i. flag_multi_class=True ii. as_gray=False iii. Something like testGene = testGenerator(test_path, flag_multi_class=True, target_size=target_size, as_gray=False)
3. Leave the other part of the code as they are. Note target_size has to be the same as the one used in trainGenerator for the model during training.

I will suggest to save the results as tif file. Modifications to saveResults fn to be madeas follows: io.imsave(os.path.join(save_path,"%d_predict.tif"%i),img_as_float(img))

[ojedaa]

I have two questions: If I work with RGB trainning images and I use binary masks. Do I have to put flag_multi_class=True? The other one, is if I trainned with the target size reescalled, and you mentioned to comment #img = trans.resize(img,target_size), how does the generator knows how to resize the test images? Thanks, O.

[yamisky]

model.py： ①def unet(pretrained_weights = None,input_size = (528, 960, 3)):
②
conv9 = Conv2D(9, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9) conv10 = Conv2D(3, 1, activation = 'sigmoid')(conv9)

data.py： ①"grayscale"→"rgb" ②#img = np.reshape(img,img.shape+(1,)) if (not flag_multi_class) else img ③#num = img[k][j] ④if img[k][j]< (threshold/255.0) → if img[k][j][0] < (threshold/255.0) and img[k][j][1] < (threshold/255.0) and img[k][j][2] < (threshold/255.0):

result:

[ghost]

Hi, is it possible to work with 4- or even 6-Channel-Images rather than with 3-Channel-RGB?

[manvirvirk]

@deaspo I made above changes as i m using rgb training images and grayscale mask still i m getting following error. Can you plz help: ValueError: Error when checking target: expected conv2d_11 to have shape (16, 16, 1) but got array with shape (256, 256, 1)

[deaspo]

target

Is it possible to convert the grayscale to "rgb"? The target has to be the same as the one used in training...

[deaspo]

I have two questions: If I work with RGB trainning images and I use binary masks. Do I have to put flag_multi_class=True? The other one, is if I trainned with the target size reescalled, and you mentioned to comment #img = trans.resize(img,target_size), how does the generator knows how to resize the test images? Thanks, O.

My comment was if you want to resize your 3 channel images, don't use the numpy resize function. There are other ways to resize/reshape a 3 channel image. See here at Stack

[deaspo]

I have two questions: If I work with RGB trainning images and I use binary masks. Do I have to put flag_multi_class=True? The other one, is if I trainned with the target size reescalled, and you mentioned to comment #img = trans.resize(img,target_size), how does the generator knows how to resize the test images? Thanks, O.

My comment was if you want to resize your 3 channel images, don't use the numpy resize function. There are other ways to resize/reshape a 3 channel image. See here at Stack

For example, I had grayscale images (or can convert to grayscale), resized them and applied stack to 3 channels again. See below for an example

img = np.resize(img,target_size)
img = np.stack((img,img,img), axis=2)

For RGB images, check the StackOverFlow

[manvirvirk]

target

Is it possible to convert the grayscale to "rgb"? The target has to be the same as the one used in training... got it!!! So what changes need to be made in the code? Thanks

[deaspo]

target

Is it possible to convert the grayscale to "rgb"? The target has to be the same as the one used in training... got it!!! So what changes need to be made in the code? Thanks

The following code you pass the input path to images you want to convert to RGB and the save path for the converted files. target size is the dimensions for the converted files and pass as gray to False. See below for example

def grayToRGBConvert(input_path, save_path, target_size = (256,256), as_gray = True):
    for entry in os.listdir(input_path):
        if os.path.isfile(os.path.join(input_path, entry)):
            if entry.lower().endswith(('.png', '.jpg', '.jpeg')):
                img = io.imread(os.path.join(input_path,entry),as_gray = as_gray)
                img = np.resize(img,target_size)
                img = np.stack((img,img,img), axis=2)
                io.imsave(os.path.join(save_path, entry), img)

And use as follows:

target_shape = (256,256)
as_gray = False
grayToRGBConvert(test_input_path_image,test_save_path_image,target_size=target_shape, as_gray=as_gray)

The line that does actual conversion is:

 img = np.stack((img,img,img), axis=2)

There are other ways, check StackOverFlow, but I use this.

[deaspo]

Probably Is I should upload the updated data.py file with all the changes for RGB model. To do that soon

[manvirvirk]

thanks for helping long way.

On Sat, Apr 11, 2020 at 1:11 AM Polycarp Okock notifications@github.com wrote:

Probably Is I should the updated data.py file with all the changes. To do that soon

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zhixuhao/unet/issues/59#issuecomment-612187239, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANAEAQACR5TW33WZQQKE4ZTRL5Y7ZANCNFSM4FNL5TIA .

[manvirvirk]

Probably Is I should upload the updated data.py file with all the changes for RGB model. To do that soon

That would be great if you do so. Thanks for helping long way.

[husheng876]

I went through the following:

1. Assuming your data is 3-channel images, else you need to convert to 3-channel images. I had only gray images and converted first to 3-channel images
2. in the fn trainGenerator, when calling it, pass the parameter as image_color_mode='rgb' and also mask_color_mode='rgb' if the mask images are also 3-channels.
3. Before proceeding, check with the dataPrepare notebook that the trainGenerator actually generates correct augmented images by setting the path for save_to_dir. By correct I mean also 3-channels as the input used.
4. If everything is okay, then the input_size to the model pass as model = unet(input_size = (X,Y,3))

The next change will be on the testGenerator fn

1. Comment this line 89 --> #img = trans.resize(img,target_size), this line always resixe the input to (x,Y,1), a one channel shape. So avoid it when your input is 3-channel
2. When calling testGenerator fn, pass the following parameters as follows: i. flag_multi_class=True ii. as_gray=False iii. Something like testGene = testGenerator(test_path, flag_multi_class=True, target_size=target_size, as_gray=False)
3. Leave the other part of the code as they are. Note target_size has to be the same as the one used in trainGenerator for the model during training.

I will suggest to save the results as tif file. Modifications to saveResults fn to be madeas follows: io.imsave(os.path.join(save_path,"%d_predict.tif"%i),img_as_float(img))

thank you for your solution. But I follow your way to change the code that does not work. And there is a mistake that the data dimensions Is not match

[HuiZhang22]

I went through the following:

1. Assuming your data is 3-channel images, else you need to convert to 3-channel images. I had only gray images and converted first to 3-channel images
2. in the fn trainGenerator, when calling it, pass the parameter as image_color_mode='rgb' and also mask_color_mode='rgb' if the mask images are also 3-channels.
3. Before proceeding, check with the dataPrepare notebook that the trainGenerator actually generates correct augmented images by setting the path for save_to_dir. By correct I mean also 3-channels as the input used.
4. If everything is okay, then the input_size to the model pass as model = unet(input_size = (X,Y,3))

The next change will be on the testGenerator fn

1. Comment this line 89 --> #img = trans.resize(img,target_size), this line always resixe the input to (x,Y,1), a one channel shape. So avoid it when your input is 3-channel
2. When calling testGenerator fn, pass the following parameters as follows: i. flag_multi_class=True ii. as_gray=False iii. Something like testGene = testGenerator(test_path, flag_multi_class=True, target_size=target_size, as_gray=False)
3. Leave the other part of the code as they are. Note target_size has to be the same as the one used in trainGenerator for the model during training.

I will suggest to save the results as tif file. Modifications to saveResults fn to be madeas follows: io.imsave(os.path.join(save_path,"%d_predict.tif"%i),img_as_float(img))

Hi，I followed all these steps, training RGB images with binary masks. But when I do the test, it gives the following error. Could you please help me to check it? Thanks

InvalidArgumentError Traceback (most recent call last)

in 2 model = unet() 3 model.load_weights("unet_membrane.hdf5") ----> 4 results = model.predict_generator(testGene,13,verbose=1) 5 saveResult("data/membrane/test",results) ~\anaconda3\lib\site-packages\keras\engine\training.py in predict_generator(self, generator, steps, callbacks, max_queue_size, workers, use_multiprocessing, verbose) 2274 'Please use `Model.predict`, which supports generators.', 2275 stacklevel=2) -> 2276 return self.predict( 2277 generator, 2278 steps=steps, ~\anaconda3\lib\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs) 65 except Exception as e: # pylint: disable=broad-except 66 filtered_tb = _process_traceback_frames(e.__traceback__) ---> 67 raise e.with_traceback(filtered_tb) from None 68 finally: 69 del filtered_tb ~\anaconda3\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 52 try: 53 ctx.ensure_initialized() ---> 54 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 55 inputs, attrs, num_outputs) 56 except core._NotOkStatusException as e: InvalidArgumentError: Graph execution error: Detected at node 'model_1/concatenate_4/concat' defined at (most recent call last): File "C:\Users\Hui\anaconda3\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Hui\anaconda3\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel_launcher.py", line 16, in app.launch_new_instance() File "C:\Users\Hui\anaconda3\lib\site-packages\traitlets\config\application.py", line 845, in launch_instance app.start() File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel\kernelapp.py", line 612, in start self.io_loop.start() File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 199, in start self.asyncio_loop.run_forever() File "C:\Users\Hui\anaconda3\lib\asyncio\base_events.py", line 570, in run_forever self._run_once() File "C:\Users\Hui\anaconda3\lib\asyncio\base_events.py", line 1859, in _run_once handle._run() File "C:\Users\Hui\anaconda3\lib\asyncio\events.py", line 81, in _run self._context.run(self._callback, *self._args) File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\ioloop.py", line 688, in lambda f: self._run_callback(functools.partial(callback, future)) File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\ioloop.py", line 741, in _run_callback ret = callback() File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\gen.py", line 814, in inner self.ctx_run(self.run) File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\gen.py", line 775, in run yielded = self.gen.send(value) File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 365, in process_one yield gen.maybe_future(dispatch(*args)) File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\gen.py", line 234, in wrapper yielded = ctx_run(next, result) File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell yield gen.maybe_future(handler(stream, idents, msg)) File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\gen.py", line 234, in wrapper yielded = ctx_run(next, result) File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 543, in execute_request self.do_execute( File "C:\Users\Hui\anaconda3\lib\site-packages\tornado\gen.py", line 234, in wrapper yielded = ctx_run(next, result) File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 306, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "C:\Users\Hui\anaconda3\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2894, in run_cell result = self._run_cell( File "C:\Users\Hui\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2940, in _run_cell return runner(coro) File "C:\Users\Hui\anaconda3\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner coro.send(None) File "C:\Users\Hui\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3165, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "C:\Users\Hui\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3357, in run_ast_nodes if (await self.run_code(code, result, async_=asy)): File "C:\Users\Hui\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3437, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 4, in results = model.predict_generator(testGene,13,verbose=1) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\training.py", line 2276, in predict_generator return self.predict( File "C:\Users\Hui\anaconda3\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\training.py", line 1982, in predict tmp_batch_outputs = self.predict_function(iterator) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\training.py", line 1801, in predict_function return step_function(self, iterator) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\training.py", line 1790, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\training.py", line 1783, in run_step outputs = model.predict_step(data) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\training.py", line 1751, in predict_step return self(x, training=False) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\base_layer.py", line 1096, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\functional.py", line 451, in call return self._run_internal_graph( File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\functional.py", line 589, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\engine\base_layer.py", line 1096, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\layers\merge.py", line 183, in call return self._merge_function(inputs) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\layers\merge.py", line 531, in _merge_function return backend.concatenate(inputs, axis=self.axis) File "C:\Users\Hui\anaconda3\lib\site-packages\keras\backend.py", line 3313, in concatenate return tf.concat([to_dense(x) for x in tensors], axis) Node: 'model_1/concatenate_4/concat' ConcatOp : Dimension 1 in both shapes must be equal: shape[0] = [1,64,64,512] vs. shape[1] = [1,32,32,512] [[{{node model_1/concatenate_4/concat}}]] [Op:__inference_predict_function_4571]