Team_SIOL

Team Members

1. Manideep Kolla (mk1723@msstate.edu) [Team Leader]
2. Abdur Rahman (ar2806@msstate.edu)
3. Amirhossein Eskorouchi (ae1028@msstate.edu)

Automated Neural Architecture Search for Optimal Model Design

This project leverages Neural Architecture Search (NAS) using AutoKeras to discover the best-performing model architecture for our task automatically. We start with a search space, utilize NAS to explore and optimize model structures, and then retrain the best-found architecture from scratch to leverage its full potential. Finally, we saved the optimized model weights and provided a script to evaluate the model on unseen test data for the AI Club Campus Vision Challenge.

Neural Architecture Search (NAS)

NAS is a technique that aims to discover optimal neural network structures, often using AutoML tools like AutoKeras. The goal is to identify high-performance architectures that outperform manually designed models through a search process that evaluates possible configurations.

The process consists of three main components:

1. Search Space: In this project, we utilized the original search space defined in the AutoKeras library. This search space included the normalization layer, augmentation layer, and feature extraction block type. On top of these, we had the classification layer that involved the choice of spatial reduction type and several dropout options. The spatial reduction types included different transformations of the features, including flattening, taking the global average or global maximum. The search space also included the choice of the optimizer and learning rate. Moreover, the augmentation layer had several options: translation, horizontal flip, vertical flip, rotation, zoom, and contrast. The most important choice in this search space is the choice of feature extraction block. The options for this block include a vanilla architecture with several convolutional, max-pooling, and dropout layers; several ResNet architectures; an Xception architecture; and several EfficientNet architectures.
2. Search Strategy: We employed Bayesian Optimization (BO) as the search strategy. Using BO, we tried to solve the following optimization problem:

                                                                                    Objective:

$$\arg\max{C{liki}} f(C{lik_i}, X, y)$$

                                                                                    Subject to:

$$\sum{i=1}^{n}\sum{k_i=1}^{Gi}C{lik_i} = 1 , \quad \forall \quad l$$

$$C_{lik_i} \in {0,1}, \quad \forall \quad l, i, k_i$$

$$k_i \in \mathbb{N}, \quad \forall \quad i$$

where $f$ is the objective function, in this case, validation accuracy, $C_{lik_i}$ denotes the $i$-th operation with $ki$-th setting is selected at $l$-th layer if $C{liki}=1$, otherwise $C{lik_i}=0$. $G_i$ is the total number of setting options for the $i$-th operation. For instance, translation_factor is an operation in the augment layer with two setting choices of 0.0 and 0.1.

3. Performance Estimation Strategy: We evaluated the best model in terms of validation accuracy. We divided the original training data into training and validation sets.

Code Usage

Installation

I used Anaconda to create a virtual environment and installed the Cuda toolkit and CuDNN using the following commands in the Anaconda prompt. I used GPU to train and evaluate the model's performance.

First, Clone the repository.

git clone https://github.com/abdurrahman1828/Team_SIOL.git
cd Team_SIOL

Then, create the virtual environment.

conda create --name ai python=3.9

Then, activate the environment.

conda activate ai

Then, install the Cuda toolkit and CuDNN:

conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0

To install the other required packages, please run the following command:

pip install -r requirements.txt

Running the NAS Script

To run the NAS on the training dataset, use the following command and specify the directories:

python nas.py --data_dir path/to/dataset --result_dir path/to/results --model_save_path path/to/best_model.keras --best_weights_file path/to/best_weights.h5

At this point, you will have a summary of the best model and the best model is saved. However, this model is not suitable for submission for competition purposes. Therefore, we developed the code for this best architecture from the model summary and used the same parameters obtained by the optimization process. In the next step, we provide the instructions for running the script to retrain this best model. This model is the best for our dataset, but it could be different for your custom dataset.

Retraining the best architecture from scratch using training data

To retrain the best model obtained in NAS using the training data, use the following command and specify the directories:

python train.py --data_dir path/to/dataset --result_dir path/to/results --best_weights_file path/to/results/best_model.weights.h5

Training and Validation Loss and Accuracy Plots:

Performance metrics on the validation set (80:20 split):

We will save the best model weights after training and use them for future evaluation of unseen data from the same label space.

Evaluating the best model on test data

To evaluate the trained model on a new test dataset, use the following command and specify the directories (you may place the best weights in the model folder:

python evaluate_model.py --test_dir path/to/test_data --weights_path path/to/best_model.weights.h5

Acknowledgements

1. AutoKeras - https://autokeras.com/ - for the NAS framework.
2. MSU AI Club - https://www.msuaiclub.com/ - for the dataset and opportunity to participate in the AI Club Campus Vision Challenge.