Getting random output while training on personal data

[gr8Adakron]

I am trying to train the model using my personal data, but the output I am getting is completely random(after saving its a pitch-black image). Sample images:

That's the input image and ground truth I am training on. As per the PASCAL dataset, there are multiple objects in the frame but for my dataset its just one object, which I have demonstrated using blue-pixel in GT.

My training command:

python train.py --max_to_keep=50 
   --Epochs=150 
  --momentum=0.9 
  --learning_rate=.0000001 
  --train_crop_size=480 
  --clip_by_value=1.0
   --train_text=data/INTERNAL/train.txt     
  --tf_initial_checkpoint=data/checkpoints/ 
  --label_dir=data/INTERNAL/ground_truth/  
  --image_dir=data/INTERNAL/JPEGImages/

1. Train dataset: 3.5K images
2. Validation dataset: 500 images
3. 150 epoch took: 24hrs on NVIDIA 2080i
4. TensorFlow-gpu==1.13.2

eval.py:

import numpy as np
import tensorflow as tf
from model_contour import build_model
slim = tf.contrib.slim
flags = tf.app.flags
from tensorflow.python.ops import variables
from utils import random_crop_and_pad_image
import os
import cv2
import time

FLAGS = flags.FLAGS
flags.DEFINE_integer('eval_crop_size', 480,
                           'Image crop size [height, width] for evaluation.')

flags.DEFINE_string('checkpoint', 'data/INTERNAL/log/model.ckpt-478439',
                    'The initial checkpoint in tensorflow format.')

flags.DEFINE_string('image_dir', 'data/INTERNAL/JPEGImages/',
                    'The Image Directory.')

flags.DEFINE_string('save_preds','create_labels/',
                    'Path to folder where predictions will be saved.')

flags.DEFINE_string('eval_text', 'data/INTERNAL/val.txt',
                    'The Path to the text file containing names of Images and Labels')###This text file should not have extensions in their names such as 8192.png or 8192.jpg instead just the name such as 8192

Image_directory = 'data/INTERNAL/JPEGImages/'
my_log_dir='./logs'

def load(saver, sess, ckpt_path):
    saver.restore(sess, ckpt_path)
    print("Restored model parameters from {}".format(ckpt_path))

image_ph = tf.placeholder(tf.uint8,[1,None,None,3],name='image_placeholder')
print("Eager executions")
print(image_ph)
size = FLAGS.eval_crop_size
print(tf.squeeze(image_ph))

image_shape = tf.shape(tf.squeeze(image_ph))
image = tf.squeeze(image_ph)
crop_h   = size
crop_w   = size
ignore_label=255
pad = tf.image.pad_to_bounding_box(image, 0, 0, tf.maximum(crop_h, image_shape[0]), tf.maximum(crop_w, image_shape[1]))    
last_image_dim = tf.shape(image)[-1]
img_crop = tf.random_crop(pad, [crop_h,crop_w,3])
img_crop.set_shape((crop_h, crop_w, 3))
print(img_crop)
print("Cropped")

image=random_crop_and_pad_image(tf.squeeze(image_ph),size,size)
norm_image = tf.image.per_image_standardization(tf.squeeze(image))
norm_image = tf.expand_dims(norm_image,dim=0)
pred = build_model(norm_image)
restore_var =  tf.trainable_variables()
pred = tf.nn.sigmoid(pred)
loader = tf.train.Saver(var_list=restore_var)

init = variables.global_variables_initializer()
with tf.Session() as sess: 
    sess.run(init)
    load(loader, sess, FLAGS.checkpoint)
    f = open(FLAGS.eval_text,'r')
    message = f.read()
    lines = message.split('\n')
    for l in lines:
        #try :    
        input_image = cv2.imread(Image_directory+l+'.jpg')
        feed_dict={image_ph:np.array([input_image])}
        P= sess.run(pred, feed_dict=feed_dict)
        print(P.shape)
        output_image = P[0]
        print(np.unique(output_image))
        print(output_image)
        print(np.where(output_image>0.9))
        cv2.imwrite("temp.jpg",output_image)
        np.save(FLAGS.save_preds+l,P)
        break;
        # except:
        #     print("ERROR")

OUTPUT:

[[[2.2268295e-04]
  [4.1663647e-05]
  [1.6093254e-06]
  ...
  [3.1292439e-06]
  [3.8594007e-05]
  [2.2462010e-04]]

 [[2.5779009e-05]
  [9.8347664e-07]
  [8.9406967e-08]
  ...
  [1.1920929e-07]
  [2.3841858e-06]
  [2.5182962e-05]]

 [[2.1159649e-06]
  [5.9604645e-08]
  [0.0000000e+00]
  ...
  [0.0000000e+00]
  [3.2782555e-07]
  [2.9504299e-06]]
 ....

 [[7.2121620e-06]
  [3.5762787e-07]
  [0.0000000e+00]
  ...
  [0.0000000e+00]
  [1.1920929e-07]
  [5.6028366e-06]]

 [[3.2663345e-05]
  [2.6524067e-06]
  [0.0000000e+00]
  ...
  [2.9802322e-07]
  [1.5497208e-06]
  [6.7502260e-05]]

 [[5.4717064e-04]
  [2.7447939e-05]
  [1.8775463e-06]
  ...
  [5.8114529e-06]
  [7.6144934e-05]
  [2.7817488e-04]]]

OUTPUT image after saving using cv2.imwrite:

[gr8Adakron]

@Raj-08 any help would be appreciated, thanks.

[innilarose]

Hello @Raj-08 even i am facing the same issue,I would like to know how to resolve it. Thanks.

[Raj-08]

@gr8Adakron @innilarose Hi guys I apologise for not being able to reply on time as I was busy with my job.

Can you guys try this

cv2.imwrite("temp.jpg",output_image*255) ?

[innilarose]

Hai @Raj-08, I tried the above and i am getting an output image like:

Is this problem related to my checkpoint file?

[Raj-08]

@innilarose can you post here your input image and input label?

[innilarose]

@Raj-08 I am training the model with custom dataset of 200 images and 200 label images. I am not using PASCAL dataset. The input image:

The label image:

I am storing the results in log directory. I am using another image for evaluation in eval.py. I have provided the checkpoint path from the checkpoint.txt file. Evaluation input image:

Output:

[Raj-08]

if you arent using pascal then create_label.py wont work for you

[Raj-08]

We will have to create contours using some sort of edge detectors. Rjusnba@gmail.com. Kindly mail me on this , i will help you out