Image Captionining for Medical Images

[FaheemParacha]

Hi Thanks a lot for giving such a great and informative tutorial. i am using your code to do the same with medical iamges you did for the natural images. I created dataset successfully. But now i stuck at a point while trining can you plz tell me what is the possible issue if you guide me it would be really so nice of you. following error is showing

---

ValueError Traceback (most recent call last)

in () 20 total_loss_test = 0 21 for (batch, (img_tensor, target)) in enumerate(test_dataset): ---> 22 batch_loss, t_loss = test_step(img_tensor, target) 23 total_loss_test += t_loss 24 # storing the epoch end loss value to plot later 7 frames /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs) 971 except Exception as e: # pylint:disable=broad-except 972 if hasattr(e, "ag_error_metadata"): --> 973 raise e.ag_error_metadata.to_exception(e) 974 else: 975 raise ValueError: in user code: :19 test_step * predictions, hidden, attention_weights = decoder(dec_input, features, hidden) :60 call * score = tf.matmul(output, self.wa(features), transpose_b=True) /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper ** return target(*args, **kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3217 matmul a, b, adj_x=adjoint_a, adj_y=adjoint_b, name=name) /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_math_ops.py:1562 batch_mat_mul_v2 "BatchMatMulV2", x=x, y=y, adj_x=adj_x, adj_y=adj_y, name=name) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper attrs=attr_protos, op_def=op_def) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py:593 _create_op_internal compute_device) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:3485 _create_op_internal op_def=op_def) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:1975 __init__ control_input_ops, op_def) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:1815 _create_c_op raise ValueError(str(e)) ValueError: Dimensions must be equal, but are 64 and 7 for '{{node rnn__global__decoder_1/MatMul}} = BatchMatMulV2[T=DT_FLOAT, adj_x=false, adj_y=true](rnn__global__decoder_1/gru_2/transpose_1, rnn__global__decoder_1/dense_16/BiasAdd)' with input shapes: [64,1,512], [7,49,512]

[SubhamIO]

@FaheemParacha [64,1,512], [7,49,512] check your dimensions in which you are converting your matrices. Matrix multiplication can only happen when dimension are correct

[FaheemParacha]

Thank you so much issue has been resolved but the results i am getting are very Bad. any suggestion what should i use for more improvement of the results in both captions as well as Blue score.

[SubhamIO]

@FaheemParacha Use larger dataset, train for more epochs, instead of beam search try to use greedy search. My beam search is under renovation :D

[FaheemParacha]

I am using Indiana Dataset which contain almost 7500 X-Rays is this considered to be a good dataset.?

[SubhamIO]

@FaheemParacha The dataset is not so large. Try using data augmentation techniques.

[FaheemParacha]

Just a quick last question. the dataset contain Normal Images and Captions too much. so with this dataset when i train this and test any image it always show the same caption. any method so i can less that normal data. becasue if i do this manually it took a lot of time. Thanks

[SubhamIO]

@FaheemParacha Things you can do : 1. shuffle the captions , 2. optimize beam search, 3. image augmentations, 4. Train for more epochs

[rohan-badugula]

Thank you so much issue has been resolved but the results i am getting are very Bad. any suggestion what should i use for more improvement of the results in both captions as well as Blue score.

How did you resolve it>