Automated Model Training and Fine-Tuning Pipeline

[devansh-shah-11]

Is your feature request related to a problem? Please describe. Training and fine-tuning models often involve significant manual work, especially when experimenting with different hyperparameters and architectures. This slows down research and model iteration.

Describe the solution you'd like Develop an automated pipeline for model training and fine-tuning that handles hyperparameter tuning and evaluation with minimal setup. The pipeline should be optimized for cloud environments like Kaggle and Colab, enabling researchers to run multiple experiments without manual intervention. Take all parameters and values from a config.yaml file.

Describe alternatives you've considered Using existing AutoML tools but they don't support customizations like different Architectures

Additional context It should support frameworks like PyTorch or TensorFlow to ensure wide usability.

Checklist

• [ ] Design the automated pipeline architecture

  • Outline key steps: dataset loading, model training, fine-tuning, hyperparameter tuning, and evaluation.

• [ ] Create scripts for automated model training

  • Ensure seamless integration in cloud environments like Kaggle/Colab.

• [ ] Incorporate model evaluation metrics

  • Refer to the custom evaluation code written and improvise on it if required.

• [ ] Automate model fine-tuning process

  • Ensure that models can be fine-tuned easily, just giving the config file using the pipeline.

• [ ] Test the pipeline in Kaggle/Colab

  • Ensure that the pipeline works end-to-end with minimal intervention.

• [ ] Document the pipeline usage

  • Provide clear instructions for researchers to use the automated training and fine-tuning pipeline.

[Sai-ganesh-0004]

Hello I would like to work on this Can you assign it to me

[devansh-shah-11]

Hey - any progress on the task? If you need any help, you can reach out to us

[Sai-ganesh-0004]

I'm going to tell what I'm going to do and tell me if I'm wrong anywhere First to implement the automated pipeline I'm going to create a config.yaml file and add necessary code in it, then I'm going to create a scripts folder in which there will be three files train.py, evaluate.py fine_tune.py and then modify main.py to execute training and evaluation. and as for Hyperparameter tuning I'm going to add necessary code into train.py and config.yaml then ill make sure to test the pipeline