[M1-Dev] Scripts to train and choose models on GPU clusters

[pab1s]

I need scripts to train different models with different hyperparameters in order to choose the smallest model that can give me better metrics. The scripts should do the following tasks:

• Find a good learning rate (I will not use schedulers for learning rate, as I expect fast convergence of the models due to prior experiments with this dataset).
• Losses. the main candidates will be: Mean Square Error Loss and
• optimizers: the main candidates will be: Adam Optimizer and Stochastic Gradient Descent.
• Batch size: will try different batch sizes ranging from mini-batch to batches up to 64 instances.
• Some other specific hyperparameters intrinsic of the model.

After this, it would be a good idea to try out different transformations to images and choose the best fitting ones. The task will consists of choosing one or two models from each family to with high performance in the classification task and do a preliminary analysis of the topology of the data.

DISCLAIMER: Some of these candidates can change or be updated. In that case, it will be discussed here.

[pab1s]

After days researching about the best way to find these parameters, I will opt to choose them automatically for data augmentation and learning rate. I will use AutoAugment: Learning Augmentation Policies from Data as strategy to select transformations on data. With respect to the learning rate, the paper Cyclical Learning Rates for Training Neural Networks mentions a strategy to choose a nice option as a learning rate. The details of these methods will be discussed in the thesis.

Moreover, the other options will be tried and see which one gives better results.