ExploreCSR Sp23 CV Notebook II

[gchochla]

Introduction to CV models:

• Conceptually, at a high level
  • [x] RNNs
  • [x] CNNs
  • [x] Transformers
• Utilities for projects
  • [x] Dataset
  • Feature extraction
  • [x] ViT
  • [x] ResNet
  • [x] ViLT
  • [x] CLIP
• Example project implementation
  • [x] Toy examples
  • [ ] @ketakilolage
  • [ ] @Aditya-1500
• [x] Links to project pages for ideas @ketakilolage @Aditya-1500

[gchochla]

When notebook is ready, I'll share link here

[gchochla]

Utilities are up and seemingly running: https://colab.research.google.com/drive/1Gpe47AP378P0tsPA3ss0E_Imazu9llvx?usp=share_link @ketakilolage @Aditya-1500

[ketakilolage]

Project Ideas

1. Image captioning
   https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning

2. Pedestrian detection
   

   • Using CNN: https://www.kaggle.com/code/wldzia/pedestrian-detection-using-cnn
   • Using ViT: https://keras.io/examples/vision/object_detection_using_vision_transformer
     

Datasets: https://www.cis.upenn.edu/~jshi/ped_html/; https://www.kaggle.com/datasets/karthika95/pedestrian-detection

3. Google QuickDraw dataset
   https://github.com/googlecreativelab/quickdraw-dataset

4. A deep learning approach for brain tumor detection using magnetic resonance imaging
   https://arxiv.org/ftp/arxiv/papers/2210/2210.13882.pdf
   "an architecture containing five convolution layers, five max-pooling layers, a Flatten layer, and two dense layers has been proposed for detecting brain tumors from MRI images. The proposed model includes an automatic feature extractor, modified hidden layer architecture, and activation function."

[gchochla]

Great! Try to implement one of these using the given utilities in the notebook. Example usage in "Examples of utilities" section. The notebook was set up to extract feature vectors from images (+ text optionally). You'll have to set up utilities on top of that for more intricate projects. Try to use sklearn on top of extracted features if possible

[ketakilolage]

Another repo for image captioning. This one doesn't mention the pytorch version, but I believe it should be a recent one since the repo is from 2022. https://github.com/amirhosein-mesbah/Deep_Learning/tree/main/Image_Captioning

I believe students would be able to change and replace anything outdated. The first repo is still better from a learning pov.