search

greydanus / mnist1d

A 1D analogue of the MNIST dataset for measuring spatial biases and answering Science of Deep Learning questions.
Apache License 2.0
199 stars 33 forks source link
convnet dataset machine-learning neural-networks pytorch research

readme

The MNIST-1D Dataset

ICML 2024 | Blog post | Paper at arXiv | Paper at OpenReview | GitHub

Most machine learning models get around the same ~99% test accuracy on MNIST. Our dataset, MNIST-1D, is 100x smaller (default sample size: 4000+1000; dimensionality: 40) and does a better job of separating between models with/without nonlinearity and models with/without spatial inductive biases.

Dec 5, 2023: MNIST-1D is now a core teaching dataset in Simon Prince's Understanding Deep Learning textbook

Citation:

@inproceedings{greydanus2024scaling,
  title={Scaling down deep learning with {MNIST}-{1D}},
  author={Greydanus, Sam and Kobak, Dmitry},
  booktitle={Proceedings of the 41st International Conference on Machine Learning},
  year={2024}
}

overview.png

Quickstart and use cases

Installing with pip

pip install mnist1d

This allows you to build the default dataset locally:

from mnist1d.data import make_dataset, get_dataset_args

defaults = get_dataset_args()
data = make_dataset(defaults)
x, y, t = data['x'], data['y'], data['t']

If you want to play around with this, see notebooks/mnist1d-pip.ipynb.

Alternatively, you can always pip install via the GitHub repo:

python -m pip install git+https://github.com/greydanus/mnist1d.git@master

Comparing MNIST and MNIST-1D

Dataset Logistic regression MLP CNN GRU* Human expert
MNIST 94% 99+% 99+% 99+% 99+%
MNIST-1D 32% 68% 94% 91% 96%
MNIST-1D (shuffle**) 32% 68% 56% 57% ~30%

*Training the GRU takes at least 10x the walltime of the CNN.

**The term "shuffle" refers to shuffling the spatial dimension of the dataset, as in Zhang et al. (2017).

According to Geoffrey Hinton, the original MNIST dataset is the Drosophila of machine learning. But we argue that it has a few drawbacks:

Dimensionality reduction

Visualizing the MNIST and MNIST-1D datasets with t-SNE. The well-defined clusters in the MNIST plot indicate that the majority of the examples are separable via a kNN classifier in pixel space. The MNIST-1D plot, meanwhile, reveals a lack of well-defined clusters which suggests that learning a nonlinear representation of the data is much more important to achieve successful classification.

Downloading the dataset

Here's a minimal example of how to download the frozen dataset. This is arguably worse than installing this repo with pip and generating it from scratch. But it does have its uses. It can also be used for double-checking that the procedurally generated dataset exactly matches the one used in the paper and blog post:

from urllib.request import urlopen
import pickle

url = 'https://github.com/greydanus/mnist1d/raw/master/mnist1d_data.pkl'
data = pickle.load(urlopen(url))

data.keys()

>>> dict_keys(['x', 'x_test', 'y', 'y_test', 't', 'templates'])  # these are NumPy arrays

Constructing the dataset

This is a synthetically-generated dataset which, by default, consists of 4000 training examples and 1000 testing examples (you can change this as you wish). Each example contains a template pattern that resembles a handwritten digit between 0 and 9. These patterns are analogous to the digits in the original MNIST dataset.

Original MNIST digits

mnist1d_black.png

1D template patterns

mnist1d_black.png

1D templates as lines

mnist1d_white.png

In order to build the synthetic dataset, we pass the templates through a series of random transformations. This includes adding random amounts of padding, translation, correlated noise, iid noise, and scaling. We use these transformations because they are relevant for both 1D signals and 2D images. So even though our dataset is 1D, we can expect some of our findings to hold for 2D (image) data. For example, we can study the advantage of using a translation-invariant model (eg. a CNN) by making a dataset where signals occur at different locations in the sequence. We can do this by using large padding and translation coefficients. Here's an animation of how those transformations are applied.

mnist1d_tranforms.gif

Unlike the original MNIST dataset, which consisted of 2D arrays of pixels (each image had 28x28=784 dimensions), this dataset consists of 1D timeseries of length 40. This means each example is ~20x smaller, making the dataset much quicker and easier to iterate over. Another nice thing about this toy dataset is that it does a good job of separating different types of deep learning models, many of which get the same 98-99% test accuracy on MNIST.

Example use cases

Quantifying CNN spatial priors

For a fixed number of training examples, we show that a CNN achieves far better test generalization than a comparable MLP. This highlights the value of the inductive biases that we build into ML models.

Finding lottery tickets

We obtain sparse "lottery ticket" masks as described by Frankle & Carbin (2018). Then we perform some ablation studies and analysis on them to determine exactly what makes these masks special (spoiler: they have spatial priors including local connectivity). One result, which contradicts the original paper, is that lottery ticket masks can be beneficial even under different initial weights. We suspect this effect is present but vanishingly small in the experiments performed by Frankle & Carbin.

lottery.png

lottery_summary.png

Observing deep double descent

We replicate the "deep double descent" phenomenon described by Belkin et al. (2018) and more recently studied at scale by Nakkiran et al. (2019).

Metalearning a learning rate

A simple notebook that introduces gradient-based metalearning, also known as "unrolled optimization." In the spirit of Maclaurin et al (2015) we use this technique to obtain the optimal learning rate for an MLP.

metalearn_lr.png

Metalearning an activation function

This project uses the same principles as the learning rate example, but tackles a new problem that (to our knowledge) has not been tackled via gradient-based metalearning: how to obtain the perfect nonlinearity for a neural network. We start from an ELU activation function and parameterize the offset with an MLP. We use unrolled optimization to find the offset that leads to lowest training loss, across the last 200 steps, for an MLP classifier trained on MNIST-1D. Interestingly, the result somewhat resembles the Swish activation described by Ramachandran et al. (2017); the main difference is a positive regime between -4 and -1.

metalearn_afunc.png

Benchmarking pooling methods

We investigate the relationship between number of training samples and usefulness of pooling methods. We find that pooling is typically very useful in the low-data regime but this advantage diminishes as the amount of training data increases.

pooling.png

Dependencies