5318 Assignment todo

[henryliangt]

Visualize, calculate, code.

density based classifier is outlier friendly.

[henryliangt]

KNN - subset selection

[henryliangt]

[henryliangt]

The code to load the training and test sets is provided in the .ipynb template. For further information on the dataset, you may be interested in the following link: https://github.com/zalandoresearch/fashion-mnist Algorithm design and setup You will be required to design and implement three algorithms that we have covered in the course using the sklearn and/or keras libraries, in order to investigate their strengths and weaknesses.

1. An appropriate algorithm of your choice from the first 6 weeks of the course
2. A fully connected neural network
3. A convolutional neural network Due to runtime constraints, you will not be required to thoroughly tune over all possible neural network designs, however you should justify your design decisions in the report, and you may wish to conduct a few rough experiments to converge on a reasonable design. Remembering that the focus of the assignment is to demonstrate your understanding of and compare the algorithms, not to achieve state-of-the-art performance, keep your models appropriately small such that you can complete the hyperparameter tuning in a reasonable time on your hardware of choice. You will need to choose an appropriate number of epochs to use for your search, or use a stopping criterion. Hyperparameter tuning You may use cross validation via sklearn to search over different parameter combinations, or you may implement another reasonable search approach if you find the runtime of CV prohibitive (for example, a holdout procedure with the validation set). Please ensure you search over the following:
4. Algorithm of your choice from the first 6 weeks: • You will need to choose an appropriate set of hyperparameters to search over based on what was covered in the tutorials or other references
5. Fully connected neural network: • At least three learning rate values • At least two appropriate activation functions for the hidden layers, as well as the case where no activation function is used
6. Convolutional neural network: • At least three learning rate values • At least two kernel sizes • At least two strides Keep a record of the runtimes and results with each parameter combination (if using sklearn, consult the documentation) and use these to produce appropriate visualisations/tables of the trends in your hyperparameter search to aid the discussion in your report. In your submission, keep the parameter search separate from other cells, and submit the .ipynb with the output preserved, as markers will not be able to run this code (it will take too long)

[henryliangt]

Final models After selecting the best set of hyperparameters for each model, include cells which train the models with the selected hyperparameters independently of the parameter search cell, and measure the performance of each model on the test set.

2. Report An approximate outline of the report requirements is provided below, but make sure to reference the marking criteria also for the mark distribution. You may write your report in MS Word or LaTeX, but please ensure that it is well-formatted and submit it as a pdf. Introduction State the aim of your study. Outline the importance of your study. Even though this particular dataset may not be the most practically useful, consider the importance of comparing algorithms and their suitability for a given task, as well as the importance of image classification more broadly. Data In this section, you should describe the dataset and pre-processing. Data description and exploration. Describe the data, including all its important characteristics, such as the number of samples, classes, dimensions, and the original source of the images. Discuss your data exploration, including potential difficulties for classification such as classes with similar features and differences in pixel intensities between samples. Where relevant, you may wish to include some sample images to aid this discussion. Pre-processing. Justify your choice of pre-processing either through your insights from the data exploration or with reference to other sources. Explain the effect/purpose of any pre-processing techniques you have applied. If you have not performed pre-processing or have intentionally omitted possible steps such as image centring, justify these decisions. Methods In this section, you should explain the classification methods you used. Summary. Briefly describe the main ideas of the three algorithms - your chosen algorithm, MLP and CNN. Explain why you chose your algorithm. Strengths and weaknesses. Describe the relative strengths and weaknesses of the algorithms from a theory perspective. Consider factors such as performance, overfitting, runtime, number of params and interpretability. Explain the reasons; e.g. don’t simply state that CNNs perform better on images, but explain why this is the case. Architecture and hyperparameters – State and explain the chosen architectures or other relevant design choices, the hyperparameters you aim to tune and outline your search method. Briefly explain what each hyperparameter controls and how changing it may affect the algorithm. For example, consider the effects of changing the learning rate, or changing the stride of a convolutional layer.

[henryliangt]

Results and discussion In this section, you should present and discuss your results. Begin with the hyperparameter tuning results. Include appropriate tables or graphs (not code output) to illustrate the trends (performance, runtime etc.) across different hyperparameter values. Discuss the trends and provide possible explanations for their observation. Did they align with your predictions? Next, present a table showing the best hyperparameter combination for each algorithm, its performance on the test set (e.g. accuracy and other performance measures), and the training runtime. Analyse and discuss the results. Refer to the stated strengths and weaknesses of the classifiers in the Methods section; did the results agree with your expectations? What factors influenced the runtime (time per epoch, total number of epochs required etc.)? Include anything you consider interesting and/or relevant. For example, you may look at which classes each algorithm confused. Conclusion Summarise your main findings, mention any limitations, and suggest future work. When making your conclusions, consider not only accuracy, but also factors such as runtime and interpretability. Ensure your future work suggestions are concrete (eg. not in the spirit of “try more algorithms”) and justify why they would be appropriate. Reflection Write one or two paragraphs describing the most important thing that you have learned while completing the assignment

[henryliangt]

[henryliangt]