johko / computer-vision-course

This repo is the homebase of a community driven course on Computer Vision with Neural Networks. Feel free to join us on the Hugging Face discord: hf.co/join/discord
MIT License
376 stars 124 forks source link

Unit-3 Chapter-3 Vision Transformers Image Classification with LoRA #108

Closed alanahmet closed 6 months ago

alanahmet commented 7 months ago

Hey everyone,

This is draft for the LoRA Image Classification notebook. I would appreciate any feedback!

review-notebook-app[bot] commented 7 months ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

ricard-inho commented on 2023-12-11T04:46:06Z ----------------------------------------------------------------

I think is a little bit too wordy, I would simplify the sentence a little like:

To prepare the inputs for the model, we have to apply the required transformations. This can be achieved by utilizing the AutoImageProcessor module, which loads the appropriate transformations corresponding to the relevant model. We can see which transformations are used in the processor config.

review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

ricard-inho commented on 2023-12-11T04:46:07Z ----------------------------------------------------------------

I would simplify the first sentence to:

We can create a function which will preprocess the batch. The trainer will call this function when we add it to the dataset using with_transform during training time.

review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

ricard-inho commented on 2023-12-11T04:46:08Z ----------------------------------------------------------------

We also create collate_fn function to define how individual samples extracted from a dataset are combined into batches. collate_fn is used when iterating through the dataset in the training loop to prepare batches of data that can be fed into the model.


review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

ricard-inho commented on 2023-12-11T04:46:08Z ----------------------------------------------------------------

To keep the consistency add the code format to get_peft_model

Next, we use get_peft_model to wrap the base model so that “update” matrices are added to the respective places.

review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

merveenoyan commented on 2023-12-11T14:38:30Z ----------------------------------------------------------------

At the last line, you are installing the the ones in first line so there's no need. also, installing PEFT from main might result in problems later on, so maybe you could install the stable pypi distribution like the others.


review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

merveenoyan commented on 2023-12-11T14:38:31Z ----------------------------------------------------------------

This is mainly done for model training as labels already exist for each sample, so you can carry this before the model training section.


review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

merveenoyan commented on 2023-12-11T14:38:31Z ----------------------------------------------------------------

You could have the label2id function right before this one 😊


review-notebook-app[bot] commented 7 months ago

View / edit / reply to this conversation on ReviewNB

merveenoyan commented on 2023-12-11T14:38:32Z ----------------------------------------------------------------

You could also give a link to model repository so people can check the automatically generated card and how the training run looks like on tensorboard :)


alanahmet commented on 2023-12-11T18:04:17Z ----------------------------------------------------------------

Hi Merve, thanks for the great suggestions :)

I couldn't find the Metric tab on hub for peft model, should I also download the main model to the hub to see metrics?

merveenoyan commented on 2023-12-12T13:14:55Z ----------------------------------------------------------------

Ah no, you mean tensorboard?

alanahmet commented on 2023-12-13T08:29:48Z ----------------------------------------------------------------

Am I mistaken or do you mean a tensorboard similar to the one shown on this page? https://huggingface.co/docs/hub/tensorboard

If its the case I couldn't find metric tab on trained peft model(https://huggingface.co/alanahmet/vit-base-patch16-224-finetuned-lora-oxfordPets)

merveenoyan commented on 2023-12-13T14:04:16Z ----------------------------------------------------------------

interesting, you can ignore this for now, I'll check since it seems like a bug