Transfer Learning: Image Classification

[shreydan]

This is my setup for transfer learning:

• model: google/vit-base-patch16-224: this is the most downloaded and well-known base transformer model.
• dataset: oxford-iiit pets dataset: hf-link - quite a popular dataset for image classification on the HF hub. 37 classes with some level of imbalance, so good for comparison. Seems like a good starting point specifically for transfer learning.
• For starters, I'm sticking to using everything from the HuggingFace ecosystem: datasets, evaluate, transformers for model + training.
• I froze almost all params in the model except for the last classifier head, which is around 28k params only
• I also want to use timm & raw pytorch for everything, everything without HF, including preparing the dataset from scratch. Would appreciate your thoughts on this, on how to balance what % is HF and what is not.

[shreydan]

this is the notebook I've come up with: vit-transfer-learning-image-classification

[shreydan]

I think I need to update the folder structure, all the files should be under unit-3 only afaik

[Anindyadeep]

I think I need to update the folder structure, all the files should be under unit-3 only afaik

Can you push a PR for this, so it will be easier for us to review.

[shreydan]

discussions:

• more info on the folder structure. Currently it's this:

  .
  └── course/
  └── chapters/
      ├── chapter3/
      │   ├── Transfer Learning and Fine-tuning.mdx
      │   └── TransferLearningInDepth.ipynb
      ├── chapter-n/
      │   └── ...mdx
      └── _toctree.yml

but since we have the following sections: transfer learning, fine-tuning, fine-tuning with LoRA, knowledge distillation. We cannot fit the entirety into a single .mdx So I think each chapter should be a folder under the unit.

.
└── course/
    └── chapters/
        ├── chapter3/
        │   ├── Transfer Learning and Fine-tuning/
                        └── transfer-learning.mdx
                        └── fine-tuning.mdx
                        └── knowledge-distillation.mdx
        ├── chapter-n/
        │   └── ...mdx
        └── _toctree.yml

• Clarification on the tasks: classification (only?) or we'll also work on: segmentation, object detection.
• should it be more HF oriented or raw PyTorch oriented? Considering HF docs are pretty excellent to learn about fine-tuning and all.

Would appreciate everyone's thoughts on this: @sezan92 @Anindyadeep @alanahmet

[Anindyadeep]

Yeah that makes quite a lot of sense, we can actually go for this.