puzzles about this fancy paper

[brisker]

1. In the paper you mentioned the model can generate both image and text attention, with or without text input. So if input is only image without text, did you mean you can generate text report with text attention??
2. In this t-SNE visualization(Fig. 4), image what does "w/o text" and "w/ text" mean? w/o or w/ in training stage or in testing stage? (your work seems more fancy if it means training stage, since training with text and input without text seems more fancy, and input text seems unreasonable...)
3. If testing without text, what difference is the testing model with the training model(which is with text input)?

[zizhaozhang]

1. The text attention is meaningful when text input is given.
2. For most experiments, training is done given by both image and text. w/o text and w/ text means the situation in the testing stage.
3. No model difference, just do not send the text input (by setting the text feature matrix to zero).

[brisker]

@zizhaozhang I am very interested in this paper. Hope you can provide some answers for my questions, thanks a lot!

1. so one of the idea this paper wants to tell us is that , using both image and text to train a model, and the image classification accuracy will increase? What reason can explain this accuracy increase?

2. it seems that your report can be summarized as 5 2-class or 3-class classification problems. So in the real medical report, I wonder that, does the doctor write sentences like "moderate crowding of the nuclei can be seen" , or write a crowding label(for example, 1 indicates mild crowding, 2 indicates moderate crowding, 3 indicates severe crowding)? If write labels, why not regard this a pure image classification problem?( I mean 5 classification problems , regarding the 5 cell appearances)

3. Besides, I observed that the descriptions are mainly about the cell appearance features, so why not directly using cell segmentation results to get the cell appearance features, since this seems more reasonable and reliable than learning from the report? (Because the cell segmentation results can not be integrated into a deep learning framework?)

[zizhaozhang]

Thanks !

1. The introduced text knowledge
2. Please note that we are not trying to solve the problem of this dataset. Our model is designed to input free text. For pathology images, it is true for pathologist to write this kind of report. They prefer to not change too much. You can find evidence from a pathology book.
3. It is a fundamental logic question. Do you think image recognition is useful? Why we just do object detection?
   Pls ask further paper question using email. This is for code issue :)

[brisker]

@zizhaozhang I have already emailed you(zizhao@cise.ufl.edu) for further questions. Waiting for your answer. Thanks!