Meetings minutes

[mxochicale]

Timeline of master dissertation

Tue 7 Feb 12:30:35 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

Updates

• XZ
  • overleaf template https://www.overleaf.com/project/63e14ed20861d600c741aa3a
  • ...

    General discussion and comments

• Discuss timeline for presentation and submission of proposal on 26-April-2023

• Road map, next steps and actions

• [x] ...
• [x] Review papers on amir: https://github.com/budai4medtech/amir/tree/main/docs/references
• [x] Replicate https://medium.com/analytics-vidhya/automatic-medical-report-generation-from-x-ray-images-through-ai-fd04de21e0e5 https://github.com/budai4medtech/amir/issues/2
• [x] ...

  Pending

• [x] #3
• ...

If you feel I left something out, please add/edit it.

Thanks Miguel

[mxochicale]

Tue 14 Feb 12:30:02 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

Updates

• adds sections to the template https://www.overleaf.com/project/63e14ed20861d600c741aa3a
• https://github.com/budai4medtech/amir/issues/2
• ...

  General discussion and comments

  • sections of your dissertations
  • references
  • ...

    Road map, next steps and actions

  • [x] Keep polishing draft
  • [x] Fix this issues : https://github.com/budai4medtech/amir/issues/2
  • [x] Discuss presentation
  • [ ]

    Pending

  • [x] #3
  • ...

If you feel I left something out, please add/edit it.

Thanks Miguel

[mxochicale]

Tue 21 Feb 13:02:49 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

Updates

• dissertation draft https://www.overleaf.com/project/63e14ed20861d600c741aa3a
• replication of notebooks, and new literature
• ...

  General discussion and comments

  • Replication of https://github.com/vysakh10/Medical-Report-Generation-using-Deep-Learning
  • The use of Diffusion Models for AMIR
  • New references https://github.com/budai4medtech/amir/issues/6
  • Presentation

Road map, next steps and actions

• [x] Keep polishing draft, add more details on your method section (add draft ideas)
• [x] MX to review draft
  • added comments and few changes to Aims chapter: https://www.overleaf.com/project/63e14ed20861d600c741aa3a
• [x] Fix this issues : https://github.com/budai4medtech/amir/issues/2
• [x] Prepare draft presentation
  • Title, name, MSc course (main slide)
  • Aims (one primary aim, secondary aims (one or two))
  • Methods
  • Gantt chart: https://www.google.co.uk/search?q=Gantt+chart+phd+thesis&source=lnms&tbm=isch&sa=X&ved=2ahUKEwiGnOqh2qb9AhWZHcAKHSjWApEQ_AUoAXoECAEQAw&biw=1920&bih=1104
  • References
• [ ]

  Pending

• [x] #3
• @mxochicale Update for presentation (I just received the details required for a presentation from Holger) :
  • Presentation is 5 minutes
  • No-slide and talk cover (a) Title, (b) Introduction, (c) Methods, (d) Results, (e) Discussion, and (f) Conclusion/Summary.
  • Use the proposal template to identify the key criteria of the talk. (The template is the same as the draft proposal, I have pasted it in my draft proposal.)
  • I plan to give up ppt and write the presentation paragraph only. I will present it to you next Tuesday.

If you feel I left something out, please add/edit it.

Thanks Miguel

[mxochicale]

Tue 28 Feb 12:30:29 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Rehearsal of 5 minutes presentation. Try to practice presentation
• Replication of https://github.com/vysakh10/Medical-Report-Generation-using-Deep-Learning
• New references https://github.com/budai4medtech/amir/issues/6

Road map, next steps and actions

• [x] Prepare 5 minutes talk
• [x] Improve the aims section of your dissertation
• [x] Review other models and the semantic aliment frameworks: https://scholar.google.com/scholar?cites=12782711326953406821&as_sdt=2005&sciodt=0,5&hl=en
• [x] MX to add references for protocols
• [x] Keep polishing draft, add more details on your method section (add draft ideas)
• [x] Keep replicating https://github.com/vysakh10/Medical-Report-Generation-using-Deep-Learning
• [x] MX to look for ML tutorials

Pending

• ...

• If you feel I left something out, please add/edit it.

Thanks Miguel

[0-0zhuxiaoning]

Friday 10 March 12:30:29 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Finish replication of https://github.com/vysakh10/Medical-Report-Generation-using-Deep-Learning
• debug: the shape of the image feature doesn't match the shape of the input layer (finish)

Road map, next steps and actions

• [x] Improve the aims section of your dissertation
• [x] Try to replace the decoder/encoder model
• [x] Review other models and the semantic aliment frameworks: https://scholar.google.com/scholar?cites=12782711326953406821&as_sdt=2005&sciodt=0,5&hl=en
• [x] MX to add references for protocols
• [x] Keep polishing draft, add more details on your method section (add draft ideas)
• [x] MX to look for ML tutorials
• [x] Focus on fewer topics (advice from a tutor in presentation)

Pending

• ...

• If you feel I left something out, please add/edit it.

Thanks Miguel

[mxochicale]

Tue 14 Mar 13:21:59 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Add a new section in your dissertation talking about encoder-decoder architectures:
  • https://machinelearningmastery.com/encoder-decoder-recurrent-neural-network-models-neural-machine-translation/#:~:text=An%20Encoder%2DDecoder%20architecture%20was,of%20sequence%20token%20was%20reached.
  • https://d2l.ai/chapter_recurrent-modern/encoder-decoder.html
  • https://kikaben.com/transformers-encoder-decoder/
• ...

Road map, next steps and actions

• Add a new section in your dissertation talking about encoder-decoder architectures:
• Get results for BLEU score
• Review if ImageCLEF would useful for your work

Pending

Question:

• How can I use the pre-training tokenizer in huggingface?

If you feel I left something out, please add/edit it.

Thanks Miguel

[mxochicale]

Tue 14 Mar 13:21:59 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• ? Get results for BLEU score
• ? Review if ImageCLEF would useful for your work
• Discuss the aim and model selection and at least define the detailed purpose List for model selection:https://docs.google.com/spreadsheets/d/1C3F5zeNORxkIAbSJja39_uMyCraqvl2rerlhgaP-p58/edit?usp=sharing

Road map, next steps and actions

• Transformers

  • Image Captioning using PyTorch and Transformers in Python https://www.thepythoncode.com/article/image-captioning-with-pytorch-and-transformers-in-python
  • Neural machine translation with a Transformer and Keras https://www.tensorflow.org/text/tutorials/transformer
  • Image captioning with visual attention https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/image_captioning.ipynb#scrollTo=ONyjuWsmZoyO

• Dissertation

  • Methods (...)
  • Datasets (...) https://arxiv.org/pdf/2101.07299.pdf
  • Evaluation measures (BLEU, ROUGE-L, METEOR, CIDEr, SPICE, etc) https://doi.org/10.1016/j.patcog.2021.107856

• MIUA2023

  • https://github.com/budai4medtech/amir/issues/7

Pending

Question:

• How can I use the pre-training tokenizer in huggingface?

If you feel I left something out, please add/edit it.

Thanks Miguel

[0-0zhuxiaoning]

Tue 29 Mar 13:21:59 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Structure of results and discussion parts in dissertation.
• Image caption tutorial: how to replace coco data with IU datasets

Road map, next steps and actions

• Transformers

  • Image Captioning using PyTorch and Transformers in Python : continue try replacing dataset https://www.thepythoncode.com/article/image-captioning-with-pytorch-and-transformers-in-python
  • Rewrite the aim and intro part. Keep subsections in identical lengths.

• Dissertation

  • Methods (...)
  • Datasets (...) https://arxiv.org/pdf/2101.07299.pdf
  • Evaluation measures (BLEU, ROUGE-L, METEOR, CIDEr, SPICE, etc) https://doi.org/10.1016/j.patcog.2021.107856

Pending

Thanks Miguel

[0-0zhuxiaoning]

Tue 31 Mar 13:21:59 GMT 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Structure of results and discussion parts in dissertation.
• add experiment part and work flow of model

Road map, next steps and actions

• Transformers

  • Image Captioning using PyTorch and Transformers in Python : continue try replacing dataset https://www.thepythoncode.com/article/image-captioning-with-pytorch-and-transformers-in-python
  • Rewrite the aim and intro part. Keep subsections in identical lengths.

• Dissertation

• address comments in the "Experiments and Results" section,

• complete discussion section

• Look for other Master Dissertation to get an idea of the quality of UCL dissertations.

Pending

Thanks Miguel

[mxochicale]

Tue 9 May 11:11:38 BST 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Discussion about next steps for dissertation

Road map, next steps and actions

• 22th of May. Season 6: What's import for the results?
• Prepare further evaluation metrics results
• Have a look to examples of dissertation.

Pending

• Share colab notebook : https://colab.research.google.com/drive/1TBRvBg2KxPBG4JM6-6dRHclJcFGOEsiM?usp=sharing

Thanks Miguel

[mxochicale]

Tue 23 May 12:21:11 BST 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Discussion results of the marking on 24th May

Road map, next steps and actions

• Add diagram of the model, look for attention layer
• Implementing other metrics
• Check number of epoch in previous model to verify if the loss is converging
• Understand parts of the model and related them to the code implementation.
  • See Fig 9 and 10: https://link.springer.com/article/10.1007/s10462-022-10270-w;
  • https://towardsdatascience.com/using-attention-for-medical-image-segmentation-dd78825eaac6
• Potentially use different attention models (e.g., https://github.com/sooftware/attentions, https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html, https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html?highlight=attention#torch.nn.functional.scaled_dot_product_attention, https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html?highlight=attention#torch.nn.MultiheadAttention)
• Prototypes challenges
  • datasize is one challenge 4BG for glove and images with total
  • migration to Myriad

Previous discussions 16May2023:

• this blog explain briefly general arquictuture of image captioning  https://dida.do/blog/image-captioning-with-attention
• To look for potential models to implement https://paperswithcode.com/search?q_meta=&q_type=&q=medical+image+captioning

Pending

• Share colab notebook : https://colab.research.google.com/drive/1TBRvBg2KxPBG4JM6-6dRHclJcFGOEsiM?usp=sharing

Thanks Miguel

[mxochicale]

Tue 30 May 16:10:41 BST 2023

Attendees: XZ, MX; chair: MX; Apologies: ; Absent:_

General discussion and comments

• Discussion experiments with different architectures based on the attention model
• Dimensions of self attention

Road map, next steps and actions

• Adds loss plots
• Is the BLUE score and others are validated by clinicians?
  • https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=bleu+score+for+medical+image+reporting&btnG=
  • Some interesting papers to have look
    • http://jultika.oulu.fi/files/nbnfi-fe202301162915.pdf
    • https://doi.org/10.1016/j.patcog.2021.107856
    • https://link.springer.com/chapter/10.1007/978-3-031-29717-5_4
• Implementation of Evaluation metrics
• List of potential clinicians to evaluate performance of models

Pending

• Implementation of other scores

Thanks Miguel

[mxochicale]

Have a look to this references

• try other losses https://www.tensorflow.org/api_docs/python/tf/keras/losses
• Wang, Qingzhong, Jia Wan, and Antoni B. Chan. "On diversity in image captioning: Metrics and methods." IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 2 (2020): 1035-1049. https://scholar.google.com/scholar?cites=10325542674961427496&as_sdt=2005&sciodt=0,5&hl=en
• "Fig. 8. Comparison of average sentence and corpus bleu scores for both the architectures in combination with both the loss functions." from Maru, Harsh, T. S. S. Chandana, and Dinesh Naik. "Comparison of image encoder architectures for image captioning." In 2021 5th International Conference on Computing Methodologies and Communication (ICCMC), pp. 740-744. IEEE, 2021. https://scholar.google.com/scholar?cites=10262564394691415493&as_sdt=2005&sciodt=0,5&hl=en
• See other references https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=loss+functions+for+image+captioning+&btnG=
• "MEASURING DIVERSITY OF IMAGE CAPTIONS" from Wang 2022.
  • Latent Semantic Analysis (LSA): https://www.datacamp.com/tutorial/discovering-hidden-topics-python
  • we use n-gram or p-spectrum kernels [65] with LSA.

[mxochicale]

Fri 30 Jun 13:30:08 BST 2023

• Greedy search and change to beam search
• Dataset
  • with similar sentence (let's use diversity)
• Sent message to groups to ask for clinical evaluators of our model
• Potential experiments
  • Dataset A -- original
  • Dataset B -- pro-processed without repeated sentences
• Clinical evaluation
• Model link:
• Data reprocessing and image feature generating: https://drive.google.com/file/d/13M4E3NpS7XwpsLEIOEqWmXWoK36VHyww/view?usp=sharing
• Main model :https://drive.google.com/file/d/1bpAhhEX7vfe-Ro-9tNN7m3C6Kc_lPmuk/view?usp=sharing

[mxochicale]

Tue 4 Jul 12:23:32 BST 2023

• Better diversity with beamsearch than greedy search
• Review of "Transformers in medical imaging: a survey by shamshad; clinical report generation " > https://www.sciencedirect.com/science/article/pii/S1361841523000634
• RATCHET: RAdiological Text Captioning for Human Examined Thoraxes > https://github.com/farrell236/RATCHET
• https://arxiv.org/abs/2107.02104

MX: I have started with the reapplication of ratchet model here https://github.com/mxochicale/RATCHET (added on Sun 9 Jul 19:24:46 BST 2023)

[mxochicale]

Tue 11 Jul 15:20:43 BST 2023

• Test https://github.com/mxochicale/RATCHET/blob/master/model/transformer.py with our own data
• Different scores are difference that could lead to discussion
• Create a survey for Clinical feedback streamlit or anything else
• Create table with baseline and attention-based models
• Dataset is small? ;
• Reach out researchers UCL Centre for Medical Image Computing to ask for help on our model?
• Models
  • Baseline: https://drive.google.com/file/d/1bpAhhEX7vfe-Ro-9tNN7m3C6Kc_lPmuk/view?usp=sharing (image features, your notebook)

[mxochicale]

Tue 25 Jul 15:30:36 BST 2023

• References to plot loss and accuracy https://github.com/mxochicale/in2research2023/blob/15-transformer-model/scripts/models/transformer-model/timeseries_classification_from_scratch.ipynb
• Share transformer-base model
• Solving ValueError: Input 0 of layer "model_2" i>https://github.com/budai4medtech/amir/blob/11-tidying-up-repo/examples/dataprocessing-feature-extraction-enconder-decoder/encoder_decoder.ipynb

Shared documents (Zhu, Xiaoning 2023年8月1日 12:34):

• https://www.overleaf.com/project/63e14ed20861d600c741aa3a
• https://drive.google.com/file/d/12TCTkoSrUjYAY47ddm2sFT-YUupxmpk5/view?usp=sharing

[mxochicale]

Fri 4 Aug 12:17:40 BST 2023

Dissertation plans

• Fine tune model
• Make new survey
• New relevant papers for introduction, conclusions and future work > amir/issues/6

Surveys plans

• Adds survey results
• Timothy Bray: is coming back 14 of Aug 2023
• Adds surveys in Appendix

Repository

• Prepare repo to open source it

[mxochicale]

Tue 8 Aug 12:20:56 BST 2023

• How to improve performance of models (using tokenniser)
• Reporting survey results using "likert 5 survey report" > https://github.com/air4children/dei-hri2023/blob/main/article/notebooks/examples-plot-likert-chart.ipynb

[mxochicale]

Wed 15 Aug 12:00:29 BST 2023

• discussions around survey and structure of the dissertation

[mxochicale]

Tue 29 Aug 12:00:35 BST 2023

• Preparation for submission (deadline Thursday 31st of August 2023)
• Checking comments in the dissertation https://www.overleaf.com/project/63e14ed20861d600c741aa3a
• Open repo to the public