New references

[mxochicale]

This issue is to post any related references, code and data.

[0-0zhuxiaoning]

Exploring Discrete Diffusion Models for Image Captioning : https://github.com/buxiangzhiren/DDCap (text generation)

[0-0zhuxiaoning]

Model selection:

1. PubMedBERT (predict model pre-training on only medical resources trouble: how to fine-tune tokenizer )
   • Code: https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
   • Paper: https://ceur-ws.org/Vol-2936/paper-109.pdf (AEHRC CSIRO at ImageCLEFmed Caption 2021)
   • Google-citations: https://scholar.google.com/scholar?cites=1476060365132467752&as_sdt=2005&sciodt=0,5&hl=en
     2. attention-based GRU language generator model (what I am replicating now)
   • Code: https://github.com/bentrevett/pytorch-seq2seq/blob/master/2%20-%20Learning%20Phrase%20Representations%20using%20RNN%20Encoder-Decoder%20for%20Statistical%20Machine%20Translation.ipynb
   • Paper: https://ceur-ws.org/Vol-2936/paper-94.pdf
   • Google-citations: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Attention-based+CNN-GRU+Model+For+Automatic+Medical+Images+Captioning%3A+ImageCLEF+2021&btnG=
     3. Medical image captioning via generative pre-trained transformers (encoder, decoder with attention model is similar to what I have replication. Then the output of the first model will be the input of GTP-3, and GTP-3 will generate a new report based on the old report generated by the first model. )
   • Paper: https://www.nature.com/articles/s41598-023-31223-5
   • tutorial: https://www.analyticsvidhya.com/blog/2021/01/implementation-of-attention-mechanism-for-caption-generation-on-transformers-using-tensorflow/
   • tutorial; https://ankur3107.github.io/blogs/the-illustrated-image-captioning-using-transformers/
   • Architecture: The overall goal of our approach is to improve the quality of Encoder-Decoder-generated clinical records using the GPT-3 language model. Hence, the model consists of the Encoder, Decoder (LSTM) with an attention module and GPT-3.
   • how it works?
2. Neural Image Caption Generation with Visual Attention
   • paper: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention https://arxiv.org/abs/1502.03044
   • code: https://github.com/kelvinxu/arctic-captions
   • how it works?
3. A Transformer decoder model
   • Tutorial: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/image_captioning.ipynb#scrollTo=DfICM49WFpIb

[0-0zhuxiaoning]

list of models: https://docs.google.com/spreadsheets/d/1C3F5zeNORxkIAbSJja39_uMyCraqvl2rerlhgaP-p58/edit?usp=sharing

[mxochicale]

Have a look to some relevant references on transformers for report generation https://github.com/mindflow-institue/Awesome-Transformer#report-generation

[0-0zhuxiaoning]

Many thanks! It's really helpful. I have rewritten the aim and some method parts of the dissertation. If you have time, could you help me check them? 😉 link:https://www.overleaf.com/project/63e14ed20861d600c741aa3a

best wishes, Xiaoning Zhu

---

发件人: Miguel Xochicale, PhD @.> 发送时间: 2023年3月22日 20:33 收件人: budai4medtech/amir @.> 抄送: Zhu, Xiaoning @.>; Comment @.> 主题: Re: [budai4medtech/amir] New references (Issue #6)

⚠ Caution: External sender

Have a look to some relevant references on transformers for report generation https://github.com/mindflow-institue/Awesome-Transformer#report-generation

— Reply to this email directly, view it on GitHubhttps://github.com/budai4medtech/amir/issues/6#issuecomment-1480223781, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A27TNNSQG4WZN3PF4UE3GALW5NOYBANCNFSM6AAAAAAVDAX5NA. You are receiving this because you commented.Message ID: @.***>

[mxochicale]

This paper seems relevant to your work. Have a look to it

Despite the high number of machine learning models presented in the last few years for automatically annotating medical images with deep learning models, clear baselines to compare methods upon are still missing. We present an initial set of experimentations of a standard encoder-decoder architecture with the Indiana University Chest X-ray dataset. The experiments include different convolutional architectures and decoding strategies for the recurrent decoder module. The results here presented could potentially benefit those tackling the same task in languages with fewer linguistic resources than those available in English.

Cardillo, F.A. (2023). Baselines for Automatic Medical Image Reporting. In: Filipovic, N. (eds) Applied Artificial Intelligence: Medicine, Biology, Chemistry, Financial, Games, Engineering. AAI 2022. Lecture Notes in Networks and Systems, vol 659. Springer, Cham. https://doi.org/10.1007/978-3-031-29717-5_4

[mxochicale]

Liu, Zhengliang, et al. "c." arXiv preprint arXiv:2306.08666 (2023).

• https://arxiv.org/pdf/2306.08666.pdf
• https://huggingface.co/spaces/allen-eric/radiology-gpt

[0-0zhuxiaoning]

[heart] Zhu, Xiaoning reacted to your message:

---

From: Miguel Xochicale, PhD @.> Sent: Thursday, June 22, 2023 10:13:00 PM To: budai4medtech/amir @.> Cc: Zhu, Xiaoning @.>; Comment @.> Subject: Re: [budai4medtech/amir] New references (Issue #6)

⚠ Caution: External sender

Liu, Zhengliang, et al. "c." arXiv preprint arXiv:2306.08666 (2023).

• https://arxiv.org/pdf/2306.08666.pdf
• https://huggingface.co/spaces/allen-eric/radiology-gpt

— Reply to this email directly, view it on GitHubhttps://github.com/budai4medtech/amir/issues/6#issuecomment-1603379292, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A27TNNV4EBBIMUXHFBUJCMLXMS7OZANCNFSM6AAAAAAVDAX5NA. You are receiving this because you commented.Message ID: @.***>

[mxochicale]

Artificial intelligence tools in radiology practices have surged, with modules developed to target specific findings becoming increasingly prevalent and proving valuable in the daily emergency room radiology practice. The number of US Food and Drug Administration-cleared radiology-related algorithms has soared from just 10 in early 2017 to over 200 presently. This review will concentrate on the present utilization of AI tools in clinical ER radiology setting, including a brief discussion of the limitations of the technique. As radiologists, it is essential that we embrace this technology, comprehend its constraints, and use it to improve patient care. Another trend will be using NLP to create reports in a standardized format, regardless of the radiologists' dictation style, streamlining the communication process. NLP will likely analyze the report being dictated and identify relevant prior imaging, going beyond just the study headers. Additionally, automatic report generation for routine follow-up imaging may improve radiologists’ efficiency in clinical practice. AI may provide automated suggestions for follow-up imaging modality and timing, optimizing patient care. In conclusion, there has been a dramatic increase in the utilization of AI-powered tools in clinical radiology. It is time for us radiologists to embrace the technology, understand its limitations, and utilize the tool for better patient care.

Dundamadappa, Sathish Kumar. "AI tools in Emergency Radiology reading room: a new era of Radiology." Emergency Radiology (2023): 1-11. https://link.springer.com/article/10.1007/s10140-023-02154-5

[mxochicale]

Xu et. al., "ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders" [v1] Wed, 2 Aug 2023 17:59:45 UTC (2,530 KB)

https://arxiv.org/abs/2308.01317 tweet: https://twitter.com/chrisck/status/1687192083470180354

[mxochicale]

Rajpurkar, Pranav, and Matthew P. Lungren. "The Current and Future State of AI Interpretation of Medical Images." New England Journal of Medicine 388, no. 21 (2023): 1981-1990. DOI: 10.1056/NEJMra2301725 google-citations: https://scholar.google.com/scholar?cites=9504411958882111601&as_sdt=2005&sciodt=0,5&hl=en

[mxochicale]

Polina Golland presents good lines of research on applications of chest x-ray to predict pulmonary edema (mild/severe fluid in the lugs):

• Learning local representations: Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. "Multimodal representation learning via maximization of local mutual information." In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24, pp. 273-283. Springer International Publishing, 2021. https://scholar.google.com/scholar?cites=6504456436333684530&as_sdt=2005&sciodt=0,5&hl=en PDF: https://dspace.mit.edu/bitstream/handle/1721.1/143584/2103.04537.pdf?sequence=2&isAllowed=y CODE: https://github.com/RayRuizhiLiao/mutual_info_img_txt

• Multiple instance learning to build multimodal representation: Wang, Peiqi, William M. Wells, Seth Berkowitz, Steven Horng, and Polina Golland. "Using multiple instance learning to build multimodal representations." In International Conference on Information Processing in Medical Imaging, pp. 457-470. Cham: Springer Nature Switzerland, 2023. https://scholar.google.com/scholar?cites=10400163199680176530&as_sdt=2005&sciodt=0,5&hl=en PDF https://arxiv.org/pdf/2212.05561.pdf

Video is here: https://www.youtube.com/watch?v=p4duojuQmh0 Google-scholar of Polina Golland: https://scholar.google.com/citations?hl=en&user=4GpKQUIAAAAJ&view_op=list_works&sortby=pubdate