Training on own dataset

jeanpat commented 6 years ago

Hello, First, thank you for your implementation of SegCaps. I try to train on a small dataset consisting of 100 pairs of greyscaled/ 4 levels groundtruth ("masks") images:

pair-grey_mask The original dataset is available

100 pairs of images were copied in SegCaps/data/ :

from: ...data/imgs/train0000000.png and ...data/masks/train0000000.png
to : ...data/imgs/train0000099.png and ...data/masks/train0000099.png

then I tried to train secaps as follow. The given arguments are wrong, how could I modify them to train segcaps on those images?

Thank you.

(DeepFish) jeanpat@Dell-T5500:~/Developpement/SegCaps$ python3 ./main.py --test --Kfold 2 --net segcapsr3 --data_root_dir=data --loglevel 2 --which_gpus=-2 --gpus=0 --dataset mscoco17 --weights_path data/saved_models/segcapsr3/overlap_test.hdf5
Using TensorFlow backend.
INFO 2018-08-25 17:55:49,226: 
No existing training, validate, test files...System will generate it.
basename=['train0000050.png']
basename=['train0000051.png']
basename=['train0000052.png']
basename=['train0000053.png']
basename=['train0000054.png']
basename=['train0000055.png']
basename=['train0000056.png']
basename=['train0000057.png']
basename=['train0000058.png']
basename=['train0000059.png']
basename=['train0000060.png']
basename=['train0000061.png']
basename=['train0000062.png']
basename=['train0000063.png']
basename=['train0000064.png']
basename=['train0000065.png']
basename=['train0000066.png']
basename=['train0000067.png']
basename=['train0000068.png']
basename=['train0000069.png']
basename=['train0000070.png']
basename=['train0000071.png']
basename=['train0000072.png']
basename=['train0000073.png']
basename=['train0000074.png']
basename=['train0000075.png']
basename=['train0000076.png']
basename=['train0000077.png']
basename=['train0000078.png']
basename=['train0000079.png']
basename=['train0000080.png']
basename=['train0000081.png']
basename=['train0000082.png']
basename=['train0000083.png']
basename=['train0000084.png']
basename=['train0000085.png']
basename=['train0000086.png']
basename=['train0000087.png']
basename=['train0000088.png']
basename=['train0000089.png']
basename=['train0000090.png']
basename=['train0000091.png']
basename=['train0000092.png']
basename=['train0000093.png']
basename=['train0000094.png']
basename=['train0000095.png']
basename=['train0000096.png']
basename=['train0000097.png']
basename=['train0000098.png']
basename=['train0000099.png']
basename=['train0000000.png']
basename=['train0000001.png']
basename=['train0000002.png']
basename=['train0000003.png']
basename=['train0000004.png']
basename=['train0000005.png']
basename=['train0000006.png']
basename=['train0000007.png']
basename=['train0000008.png']
basename=['train0000009.png']
basename=['train0000010.png']
basename=['train0000011.png']
basename=['train0000012.png']
basename=['train0000013.png']
basename=['train0000014.png']
basename=['train0000015.png']
basename=['train0000016.png']
basename=['train0000017.png']
basename=['train0000018.png']
basename=['train0000019.png']
basename=['train0000020.png']
basename=['train0000021.png']
basename=['train0000022.png']
basename=['train0000023.png']
basename=['train0000024.png']
basename=['train0000025.png']
basename=['train0000026.png']
basename=['train0000027.png']
basename=['train0000028.png']
basename=['train0000029.png']
basename=['train0000030.png']
basename=['train0000031.png']
basename=['train0000032.png']
basename=['train0000033.png']
basename=['train0000034.png']
basename=['train0000035.png']
basename=['train0000036.png']
basename=['train0000037.png']
basename=['train0000038.png']
basename=['train0000039.png']
basename=['train0000040.png']
basename=['train0000041.png']
basename=['train0000042.png']
basename=['train0000043.png']
basename=['train0000044.png']
basename=['train0000045.png']
basename=['train0000046.png']
basename=['train0000047.png']
basename=['train0000048.png']
basename=['train0000049.png']
INFO 2018-08-25 17:55:49,245: 
Read image files...data/imgs/train0000077.png
Traceback (most recent call last):
  File "./main.py", line 276, in <module>
    main(arguments)
  File "./main.py", line 94, in main
    model_list = create_model(args=args, input_shape=net_input_shape, enable_decoder=True)
  File "/home/jeanpat/Developpement/SegCaps/utils/model_helper.py", line 29, in create_model
    model_list = CapsNetR3(input_shape, args.num_class, enable_decoder)
  File "/home/jeanpat/Developpement/SegCaps/segcapsnet/capsnet.py", line 55, in CapsNetR3
    name='deconv_cap_1_1')(conv_cap_4_1)
  File "/home/jeanpat/anaconda3/envs/DeepFish/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/jeanpat/Developpement/SegCaps/segcapsnet/capsule_layers.py", line 250, in call
    out_height = deconv_length(self.input_height, self.scaling, self.kernel_size, self.padding)
TypeError: deconv_length() missing 1 required positional argument: 'output_padding'

PS: The GPU is a GTX 960, 4Gb

$ nvidia-smi 
Sun Aug 26 10:00:11 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.51                 Driver Version: 396.51                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 00000000:03:00.0  On |                  N/A |
| 39%   29C    P8    12W / 130W |    312MiB /  4042MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

bayts commented 6 years ago

use the version of Keras 2.1.6. this solves the problem

jeanpat commented 6 years ago

Thanks,

keras 2.2.2 is installed:

$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:39:56) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import keras
Using TensorFlow backend.

In [2]: keras.__version__
Out[2]: '2.2.2'

So, let's downgrade and see ...

jeanpat commented 6 years ago

Here's the output after having dowgraded to keras 2.1.6 (tensorflow 1.10 gpu, was installed from source). It seems that I have to modify the network since the image shape is 82x80. Do the one channel masks images, with four labels (0, 1, 2, 3) have to be one-hot encoded so that their shape would be (80,82,4)?

(DeepFish) jeanpat@Dell-T5500:~/Developpement/SegCaps-Li$ python3 ./main.py --test --Kfold 2 --net segcapsr3 --data_root_dir=data --loglevel 2 --which_gpus=-2 --gpus=0 --dataset mscoco17 --weights_path data/saved_models/segcapsr3/overlap_test.hdf5
Using TensorFlow backend.
INFO 2018-08-27 15:13:54,687: 
Read image files...data/imgs/train0000077.png
__________________________________________________________________________________________________
Layer (type)                         Output Shape              Param #   Connected to             
==================================================================================================
input_1 (InputLayer)                 (None, 512, 512, 1)       0                                  
__________________________________________________________________________________________________
conv1 (Conv2D)                       (None, 512, 512, 16)      416       input_1[0][0]            
__________________________________________________________________________________________________
reshape_1 (Reshape)                  (None, 512, 512, 1, 16)   0         conv1[0][0]              
__________________________________________________________________________________________________
primarycaps (ConvCapsuleLayer)       (None, 256, 256, 2, 16)   12832     reshape_1[0][0]          
__________________________________________________________________________________________________
conv_cap_2_1 (ConvCapsuleLayer)      (None, 256, 256, 4, 16)   25664     primarycaps[0][0]        
__________________________________________________________________________________________________
conv_cap_2_2 (ConvCapsuleLayer)      (None, 128, 128, 4, 32)   51328     conv_cap_2_1[0][0]       
__________________________________________________________________________________________________
conv_cap_3_1 (ConvCapsuleLayer)      (None, 128, 128, 8, 32)   205056    conv_cap_2_2[0][0]       
__________________________________________________________________________________________________
conv_cap_3_2 (ConvCapsuleLayer)      (None, 64, 64, 8, 64)     410112    conv_cap_3_1[0][0]       
__________________________________________________________________________________________________
conv_cap_4_1 (ConvCapsuleLayer)      (None, 64, 64, 8, 32)     409856    conv_cap_3_2[0][0]       
__________________________________________________________________________________________________
deconv_cap_1_1 (DeconvCapsuleLayer)  (None, 128, 128, 8, 32)   131328    conv_cap_4_1[0][0]       
__________________________________________________________________________________________________
up_1 (Concatenate)                   (None, 128, 128, 16, 32)  0         deconv_cap_1_1[0][0]     
                                                                         conv_cap_3_1[0][0]       
__________________________________________________________________________________________________
deconv_cap_1_2 (ConvCapsuleLayer)    (None, 128, 128, 4, 32)   102528    up_1[0][0]               
__________________________________________________________________________________________________
deconv_cap_2_1 (DeconvCapsuleLayer)  (None, 256, 256, 4, 16)   32832     deconv_cap_1_2[0][0]     
__________________________________________________________________________________________________
up_2 (Concatenate)                   (None, 256, 256, 8, 16)   0         deconv_cap_2_1[0][0]     
                                                                         conv_cap_2_1[0][0]       
__________________________________________________________________________________________________
deconv_cap_2_2 (ConvCapsuleLayer)    (None, 256, 256, 4, 16)   25664     up_2[0][0]               
__________________________________________________________________________________________________
deconv_cap_3_1 (DeconvCapsuleLayer)  (None, 512, 512, 2, 16)   8224      deconv_cap_2_2[0][0]     
__________________________________________________________________________________________________
up_3 (Concatenate)                   (None, 512, 512, 3, 16)   0         deconv_cap_3_1[0][0]     
                                                                         reshape_1[0][0]          
__________________________________________________________________________________________________
seg_caps (ConvCapsuleLayer)          (None, 512, 512, 1, 16)   272       up_3[0][0]               
__________________________________________________________________________________________________
input_2 (InputLayer)                 (None, 512, 512, 1)       0                                  
__________________________________________________________________________________________________
mask_1 (Mask)                        (None, 512, 512, 1, 16)   0         seg_caps[0][0]           
                                                                         input_2[0][0]            
__________________________________________________________________________________________________
reshape_2 (Reshape)                  (None, 512, 512, 16)      0         mask_1[0][0]             
__________________________________________________________________________________________________
recon_1 (Conv2D)                     (None, 512, 512, 64)      1088      reshape_2[0][0]          
__________________________________________________________________________________________________
recon_2 (Conv2D)                     (None, 512, 512, 128)     8320      recon_1[0][0]            
__________________________________________________________________________________________________
out_seg (Length)                     (None, 512, 512, 1)       0         seg_caps[0][0]           
__________________________________________________________________________________________________
out_recon (Conv2D)                   (None, 512, 512, 1)       129       recon_2[0][0]            
==================================================================================================
Total params: 1,425,649
Trainable params: 1,425,649
Non-trainable params: 0
__________________________________________________________________________________________________
INFO 2018-08-27 15:13:57,269: 
Weights_path=data/data/saved_models/segcapsr3/overlap_test.hdf5
WARNING 2018-08-27 15:13:57,269: 
Unable to find weights path. Testing with random weights.
__________________________________________________________________________________________________
Layer (type)                         Output Shape              Param #   Connected to             
==================================================================================================
input_1 (InputLayer)                 (None, 512, 512, 1)       0                                  
__________________________________________________________________________________________________
conv1 (Conv2D)                       (None, 512, 512, 16)      416       input_1[0][0]            
__________________________________________________________________________________________________
reshape_1 (Reshape)                  (None, 512, 512, 1, 16)   0         conv1[0][0]              
__________________________________________________________________________________________________
primarycaps (ConvCapsuleLayer)       (None, 256, 256, 2, 16)   12832     reshape_1[0][0]          
__________________________________________________________________________________________________
conv_cap_2_1 (ConvCapsuleLayer)      (None, 256, 256, 4, 16)   25664     primarycaps[0][0]        
__________________________________________________________________________________________________
conv_cap_2_2 (ConvCapsuleLayer)      (None, 128, 128, 4, 32)   51328     conv_cap_2_1[0][0]       
__________________________________________________________________________________________________
conv_cap_3_1 (ConvCapsuleLayer)      (None, 128, 128, 8, 32)   205056    conv_cap_2_2[0][0]       
__________________________________________________________________________________________________
conv_cap_3_2 (ConvCapsuleLayer)      (None, 64, 64, 8, 64)     410112    conv_cap_3_1[0][0]       
__________________________________________________________________________________________________
conv_cap_4_1 (ConvCapsuleLayer)      (None, 64, 64, 8, 32)     409856    conv_cap_3_2[0][0]       
__________________________________________________________________________________________________
deconv_cap_1_1 (DeconvCapsuleLayer)  (None, 128, 128, 8, 32)   131328    conv_cap_4_1[0][0]       
__________________________________________________________________________________________________
up_1 (Concatenate)                   (None, 128, 128, 16, 32)  0         deconv_cap_1_1[0][0]     
                                                                         conv_cap_3_1[0][0]       
__________________________________________________________________________________________________
deconv_cap_1_2 (ConvCapsuleLayer)    (None, 128, 128, 4, 32)   102528    up_1[0][0]               
__________________________________________________________________________________________________
deconv_cap_2_1 (DeconvCapsuleLayer)  (None, 256, 256, 4, 16)   32832     deconv_cap_1_2[0][0]     
__________________________________________________________________________________________________
up_2 (Concatenate)                   (None, 256, 256, 8, 16)   0         deconv_cap_2_1[0][0]     
                                                                         conv_cap_2_1[0][0]       
__________________________________________________________________________________________________
deconv_cap_2_2 (ConvCapsuleLayer)    (None, 256, 256, 4, 16)   25664     up_2[0][0]               
__________________________________________________________________________________________________
deconv_cap_3_1 (DeconvCapsuleLayer)  (None, 512, 512, 2, 16)   8224      deconv_cap_2_2[0][0]     
__________________________________________________________________________________________________
up_3 (Concatenate)                   (None, 512, 512, 3, 16)   0         deconv_cap_3_1[0][0]     
                                                                         reshape_1[0][0]          
__________________________________________________________________________________________________
seg_caps (ConvCapsuleLayer)          (None, 512, 512, 1, 16)   272       up_3[0][0]               
__________________________________________________________________________________________________
mask_2 (Mask)                        (None, 512, 512, 1, 16)   0         seg_caps[0][0]           
__________________________________________________________________________________________________
reshape_3 (Reshape)                  (None, 512, 512, 16)      0         mask_2[0][0]             
__________________________________________________________________________________________________
recon_1 (Conv2D)                     (None, 512, 512, 64)      1088      reshape_3[0][0]          
__________________________________________________________________________________________________
recon_2 (Conv2D)                     (None, 512, 512, 128)     8320      recon_1[0][0]            
__________________________________________________________________________________________________
out_seg (Length)                     (None, 512, 512, 1)       0         seg_caps[0][0]           
__________________________________________________________________________________________________
out_recon (Conv2D)                   (None, 512, 512, 1)       129       recon_2[0][0]            
==================================================================================================
Total params: 1,425,649
Trainable params: 1,425,649
Non-trainable params: 0
__________________________________________________________________________________________________
INFO 2018-08-27 15:13:57,274: 
Testing... This will take some time...
Traceback (most recent call last):
  File "./main.py", line 276, in <module>
    main(arguments)
  File "./main.py", line 142, in main
    test(args, test_list, model_list, net_input_shape)
  File "/home/jeanpat/Developpement/SegCaps-Li/test.py", line 166, in test
    img_data = convert_img_data(img_data, 3)
  File "/home/jeanpat/Developpement/SegCaps-Li/utils/custom_data_aug.py", line 258, in convert_img_data
    img = img[:,:,:3]
IndexError: too many indices for array

Cheng-Lin-Li commented 6 years ago

Hi jeanpat,

Are you trying to training a model or testing a Segcaps model? The training parameter should be '--train' instead of '--test'. The program will reshape input images to resolution 512 X 512, and the model so far only support binary classification which means the program only support one channel mask. Yes, you have to one hot encoding your mask files for training.

Let me know if you still have any issue.

jeanpat commented 6 years ago

Hi Cheng-Lin-Li,

The training parameter should be '--train' instead of '--test'.

oops ! This is fixed. I modified the masks so that they have only one label: modified mask

Now I guess the arguments are still wrong: $python3 ./main.py --train --Kfold 2 --net segcapsr3 --data_root_dir=data --loglevel 2 --which_gpus=-2 --gpus=0 --dataset mscoco17 --weights_path data/saved_models/segcapsr3/first_train_100_images.hdf5

which yields:

Using TensorFlow backend.
INFO 2018-08-28 11:13:27,146: 
Read image files...data/imgs/train0000077.png
__________________________________________________________________________________________________
Layer (type)                         Output Shape              Param #   Connected to             
==================================================================================================
input_1 (InputLayer)                 (None, 512, 512, 1)       0                                  
__________________________________________________________________________________________________
conv1 (Conv2D)                       (None, 512, 512, 16)      416       input_1[0][0]            
__________________________________________________________________________________________________
reshape_1 (Reshape)                  (None, 512, 512, 1, 16)   0         conv1[0][0]              
__________________________________________________________________________________________________
primarycaps (ConvCapsuleLayer)       (None, 256, 256, 2, 16)   12832     reshape_1[0][0]          
__________________________________________________________________________________________________
conv_cap_2_1 (ConvCapsuleLayer)      (None, 256, 256, 4, 16)   25664     primarycaps[0][0]        
__________________________________________________________________________________________________
conv_cap_2_2 (ConvCapsuleLayer)      (None, 128, 128, 4, 32)   51328     conv_cap_2_1[0][0]       
__________________________________________________________________________________________________
conv_cap_3_1 (ConvCapsuleLayer)      (None, 128, 128, 8, 32)   205056    conv_cap_2_2[0][0]       
__________________________________________________________________________________________________
conv_cap_3_2 (ConvCapsuleLayer)      (None, 64, 64, 8, 64)     410112    conv_cap_3_1[0][0]       
__________________________________________________________________________________________________
conv_cap_4_1 (ConvCapsuleLayer)      (None, 64, 64, 8, 32)     409856    conv_cap_3_2[0][0]       
__________________________________________________________________________________________________
deconv_cap_1_1 (DeconvCapsuleLayer)  (None, 128, 128, 8, 32)   131328    conv_cap_4_1[0][0]       
__________________________________________________________________________________________________
up_1 (Concatenate)                   (None, 128, 128, 16, 32)  0         deconv_cap_1_1[0][0]     
                                                                         conv_cap_3_1[0][0]       
__________________________________________________________________________________________________
deconv_cap_1_2 (ConvCapsuleLayer)    (None, 128, 128, 4, 32)   102528    up_1[0][0]               
__________________________________________________________________________________________________
deconv_cap_2_1 (DeconvCapsuleLayer)  (None, 256, 256, 4, 16)   32832     deconv_cap_1_2[0][0]     
__________________________________________________________________________________________________
up_2 (Concatenate)                   (None, 256, 256, 8, 16)   0         deconv_cap_2_1[0][0]     
                                                                         conv_cap_2_1[0][0]       
__________________________________________________________________________________________________
deconv_cap_2_2 (ConvCapsuleLayer)    (None, 256, 256, 4, 16)   25664     up_2[0][0]               
__________________________________________________________________________________________________
deconv_cap_3_1 (DeconvCapsuleLayer)  (None, 512, 512, 2, 16)   8224      deconv_cap_2_2[0][0]     
__________________________________________________________________________________________________
up_3 (Concatenate)                   (None, 512, 512, 3, 16)   0         deconv_cap_3_1[0][0]     
                                                                         reshape_1[0][0]          
__________________________________________________________________________________________________
seg_caps (ConvCapsuleLayer)          (None, 512, 512, 1, 16)   272       up_3[0][0]               
__________________________________________________________________________________________________
input_2 (InputLayer)                 (None, 512, 512, 1)       0                                  
__________________________________________________________________________________________________
mask_1 (Mask)                        (None, 512, 512, 1, 16)   0         seg_caps[0][0]           
                                                                         input_2[0][0]            
__________________________________________________________________________________________________
reshape_2 (Reshape)                  (None, 512, 512, 16)      0         mask_1[0][0]             
__________________________________________________________________________________________________
recon_1 (Conv2D)                     (None, 512, 512, 64)      1088      reshape_2[0][0]          
__________________________________________________________________________________________________
recon_2 (Conv2D)                     (None, 512, 512, 128)     8320      recon_1[0][0]            
__________________________________________________________________________________________________
out_seg (Length)                     (None, 512, 512, 1)       0         seg_caps[0][0]           
__________________________________________________________________________________________________
out_recon (Conv2D)                   (None, 512, 512, 1)       129       recon_2[0][0]            
==================================================================================================
Total params: 1,425,649
Trainable params: 1,425,649
Non-trainable params: 0
__________________________________________________________________________________________________
WARNING 2018-08-28 11:13:29,656: 
Class weight file data/split_lists/train_split_0_class_weights.npy not found.
Computing class weights now. This may take some time.
100%|██████████████████████████████████████████| 45/45 [00:00<00:00, 568.18it/s]
WARNING 2018-08-28 11:13:29,750: 
Finished computing class weights. This value has been saved for this training split.
WARNING:tensorflow:Variable *= will be deprecated. Use `var.assign(var * other)` if you want assignment to the variable value or `x = x * y` if you want a new python Tensor object.
WARNING 2018-08-28 11:13:45,222: Variable *= will be deprecated. Use `var.assign(var * other)` if you want assignment to the variable value or `x = x * y` if you want a new python Tensor object.
/home/jeanpat/anaconda3/envs/DeepFish/lib/python3.6/site-packages/keras/engine/training.py:2087: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.
  UserWarning('Using a generator with `use_multiprocessing=True`'
2018-08-28 11:13:46.633383: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2018-08-28 11:13:46.633452: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: Dell-T5500
2018-08-28 11:13:46.633475: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: Dell-T5500
2018-08-28 11:13:46.633546: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.51.0
2018-08-28 11:13:46.633603: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.51.0
2018-08-28 11:13:46.633622: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.51.0
INFO 2018-08-28 11:13:50,774: 
2d_generate_train_batches
INFO 2018-08-28 11:13:50,776: 
path_to_np=data/np_files/train0000071.npz
INFO 2018-08-28 11:13:50,776: 
Pre-made numpy array not found for train0000071.
Creating now...
INFO 2018-08-28 11:13:50,783: 
2d_generate_train_batches
INFO 2018-08-28 11:13:50,785: 
path_to_np=data/np_files/train0000073.npz
INFO 2018-08-28 11:13:50,785: 
Pre-made numpy array not found for train0000073.
Creating now...

INFO 2018-08-28 11:13:50,791: 
2d_generate_train_batches
INFO 2018-08-28 11:13:50,794: 
path_to_np=data/np_files/train0000090.npz
INFO 2018-08-28 11:13:50,794: 
Pre-made numpy array not found for train0000090.
Creating now...
DEBUG 2018-08-28 11:13:50,795: STREAM b'IHDR' 16 13
DEBUG 2018-08-28 11:13:50,795: STREAM b'IDAT' 41 691
Epoch 1/20

----------------------------------------------------------------------------------------------------
Unable to load img or masks for train0000071
too many indices for array
Skipping file
----------------------------------------------------------------------------------------------------

INFO 2018-08-28 11:13:50,796: 
path_to_np=data/np_files/train0000088.npz
INFO 2018-08-28 11:13:50,796: 
Pre-made numpy array not found for train0000088.
Creating now...
DEBUG 2018-08-28 11:13:50,797: STREAM b'IHDR' 16 13
DEBUG 2018-08-28 11:13:50,797: STREAM b'IDAT' 41 733

----------------------------------------------------------------------------------------------------
Unable to load img or masks for train0000088
too many indices for array
Skipping file
----------------------------------------------------------------------------------------------------

INFO 2018-08-28 11:13:50,798: 
path_to_np=data/np_files/train0000057.npz
INFO 2018-08-28 11:13:50,798: 
Pre-made numpy array not found for train0000057.
Creating now...
DEBUG 2018-08-28 11:13:50,798: STREAM b'IHDR' 16 13
DEBUG 2018-08-28 11:13:50,799: STREAM b'IDAT' 41 627

----------------------------------------------------------------------------------------------------
Unable to load img or masks for train0000057
too many indices for array
Skipping file
----------------------------------------------------------------------------------------------------

INFO 2018-08-28 11:13:50,799: 
path_to_np=data/np_files/train0000073.npz
INFO 2018-08-28 11:13:50,799: 
Pre-made numpy array not found for train0000073.
Creating now...
DEBUG 2018-08-28 11:13:50,800: STREAM b'IHDR' 16 13
DEBUG 2018-08-28 11:13:50,800: STREAM b'IDAT' 41 661

----------------------------------------------------------------------------------------------------
Unable to load img or masks for train0000073
too many indices for array
Skipping file

While the script is running, the GPU doesn't seem to work:


:~$ nvidia-smi
Tue Aug 28 11:19:06 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.51                 Driver Version: 396.51                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 00000000:03:00.0  On |                  N/A |
| 39%   34C    P8    13W / 130W |    218MiB /  4042MiB |     13%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1146      G   /usr/lib/xorg/Xorg                           167MiB |
|    0      1658      G   compiz                                        47MiB |
|    0     12871      G   /usr/bin/nvidia-settings                       0MiB |
+-----------------------------------------------------------------------------+

The images are stored as png files:

jeanpat@Dell-T5500:~/Developpement/SegCaps/data/imgs$ ls |head
train0000000.png
train0000001.png
train0000002.png
train0000003.png
train0000004.png
train0000005.png
train0000006.png
train0000007.png
train0000008.png
train0000009.png

and

jeanpat@Dell-T5500:~/Developpement/SegCaps/data/masks$ ls |head
train0000000.png
train0000001.png
train0000002.png
train0000003.png
train0000004.png
train0000005.png
train0000006.png
train0000007.png
train0000008.png
train0000009.png

bayts commented 6 years ago

Hi jeanpat, your img shuld be shape(512,512,3) or (512,512,1) and mask (512,512,1) Chek it I use for training on own dataset this comand 'python3 ./main.py --train --data_root_dir=data --initial_lr 0.01 --loglevel 2 --Kfold 4 --loss dice --recon_wei 20 --which_gpu 2 --gpus 1'

jeanpat commented 6 years ago

ok, So I have to pad (512,512,1) images with (80,80,1) images. Thanks for the command. Would it be easy to modify the code itself to load images with shape (80,80,1 or 3) ?

bayts commented 6 years ago

Yes, change RESOLUTION in main.py and custom_data_aug.py and chek net_input_shape in main.py. I load images with shape (101,101,1) and program works is fine

Cheng-Lin-Li commented 6 years ago

Hi bayts,

Thanks for your comments. I will add more wordings on read me file for the shape of images. For the image resolutions, program should automatically scale your images to 512 x 512. Let me know if anything wrong.

bayts commented 6 years ago

Hi, Cheng-Lin-Li The program can use any image resolution. To do this, change 2 files. I checked it and the program works. I use it in the kaggle contest

Cheng-Lin-Li commented 6 years ago

It looks like a question for parameters and images formatting. I close this issue.

bashar-tom commented 6 years ago

can I use train using RESOLUTION other than 512 ? changing the RESOLUTION in main.py and custom_data_aug.py to 300 leads to ( A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 76, 76, 8, 32), (None, 75, 75, 8, 32)] ) i want to use smaller RESOLUTION to be able to train on a smaller GPU

Cheng-Lin-Li commented 6 years ago

Hi bashar-tom,

No. There is no parameter to change the model size. The program can only rescale your image size to fit 512x512. You have to modify the model to fit your requirements.

bashar-tom commented 6 years ago

Thank you Cheng-Lin-Li for your answer and for your remarkable work

I am doing a master thesis on object contour detection and I have used contour images as ground truth I have adapted the 2d data loader according to my requirements and used bce_dice loss. Until now the model is trained for 1 Epoch but still getting an output in direction of segmentation the whole object. Could you give me any hint to proceed my work?

Here is an example of my ground truth images ex_train_after_gray2007_000068

Cheng-Lin-Li commented 6 years ago

Hi bashar-tom,

I have no experience of object contour detection. I am not really sure the SegCaps is a good candidate on this task or not. I can only suggest you overfitting on one single image with the model. If the model cannot even overfit on one single image, maybe you need to research STOA papers and learn from that. If the model can overfit on one single image, then you can try to scale it up.

Kr, Cheng-Lin Li

Cheng-Lin-Li / SegCaps

Training on own dataset #4