search

ryanchankh / redunet_demo

80 stars 11 forks source link

[question] Training on high dim dataset #1

Closed kirk86 closed 3 years ago

kirk86 commented 3 years ago

Hi, thanks for sharing this. If you don't mind I'd like to ask regarding training rotational invariance on MNIST. From my understanding you didn't train on the whole dataset, you trained on a single class/digit and then tested on another, right?

The dim of MNIST is 784, did you train using all 784 dim in data X? e.g.

net = ReduNetVector(num_classes=10, data_dim=784, num_layers=2000, eta=0.5, eps=0.1)
Z_train = net.init(X, y)

For instance on CIFAR, I've tried reducing the dim from 3072 to 192 by obtaining feature map from a convnet and finally calling

net = ReduNetVector(num_classes=10, data_dim=192, num_layers=2000, eta=0.5, eps=0.1)
Z_train = net.init(feature_map, y)

But as you can see the error is still quite high, any ideas why?

image

ryanchankh commented 3 years ago

I trained on a number of samples from each class, but data from all classes are in the same training array. (for example, 100 samples from each class, then X_train has 1000 samples.

I think a good way to evaluate the performance of your model is by looking at the heatmap of the final output representationss, Z @ Z.T. Since we are expecting subspace structure, a well-learned ReduNet should give us block diagonals in our heatmap. And metrics using nearest subspace classifier should give us good performance. kNN can technically also work, but it doesn't rely on subspace structure, hence kNN is probably not a good indicator of subspace structures in our representations.

Also, for rotational invariant MNIST, I think ReduNet1D might be more appropriate. https://github.com/ryanchankh/redunet_demo/blob/b588f6b2e279155bdc9c0c77ee9bd789c2824d4f/redunet/redunet_block.py#L25 To use this, you should convert MNIST to multi-channel 1D arrays. For the mathematical motivation behind this, feel free to check out our paper. https://arxiv.org/abs/2010.14765

Hope this helps!