Experiment 1 (robustness to the noise)

[nperraud]

Specifications:

• Nside=1024
• Smoothing 1arcmin
• Dataset: same psd (We have now 20 signals of each class)
• compare 3 different levels of noise

Expected outcome: the noise will smooth the histogram and is likely to make it fail as a classifier. However, the neural network should be quite robust to this perturbation.

[nperraud]

@mdeff, I will make the first notebook with the experiment setting. It might be tomorrow though.

[mdeff]

Great thanks @nperraud. I too have the intuition the graph CNN will work in a noisy situation (I've observed it multiple times).

[tomaszkacprzak]

Looks like a great plan! Looking forward to seeing the results. Let me know if there is anything I can do to help.

[nperraud]

So I picked up 3 different noise values: (0.05, 0.1, 0.2) and I tried it. I am getting so-so results. It needs a bit more of parameters tuning. I am actually amazed how good the histogram features are. @mdeff, you can directly take the notebooks from my branch. I fixed the seed to have reproducible results. https://github.com/SwissDataScienceCenter/scnn/blob/nati/part_sphere_whole_data-noise-01.ipynb https://github.com/SwissDataScienceCenter/scnn/blob/nati/part_sphere_whole_data-noise-02.ipynb

[mdeff]

Thanks! :+1:

I am actually amazed how good the histogram features are.

So maybe the question is: when do they fail? @tomaszkacprzak any idea?

One remark @nperraud: don't use np.random.seed(0) to fix the seed, as it fixes it for the whole numpy, and some algorithms depend on true randomness. Prefer instead rs = np.random.RandomState(0) then use e.g. rs.uniform() where you need controlled randomness.

[nperraud]

Thanks for the tips. I have changed the way I fixed the seed in the experiments.

Here are my results (They might get a little bit better). We seem to do always slightly better than the histogram. But nothing amazing.

Noise	Histogram	SCNN
0	0%	0%
0.05	0.12%	0%
0.1	6.19%	3.7%
0.2	37.50%	34.5%

Should we try with a noise level of 0.15?

I will push the nootbooks on the result branch when I will be a bit more sure about the parameters.

[mdeff]

Thanks @nperraud! It's true that the performance gap is not incredible. :(

Two ideas:

• Force the SCNN to be agnostic to the noise by adding random noise during training as a data augmentation step. (As is done for denoising auto-encoders.)
• Try other types of noise?

[nperraud]

I just pushed the results on the result branch. Here are the latest results. I think it is fine. For 0.1, the difference is significant and for 0.2, the problem probably does not make a lot of sense. Hence we should just run it again for 0.15. Then we should be good for this experiment.

Noise	Histogram	SCNN
0	0%	0%
0.05	0.12%	0%
0.1	6.19%	2.68%
0.2	37.50%	34.76%

@tomaszkacprzak, should we try another experiment?

[nperraud]

After discussion with Tomek, here are a few things to try: 1) Increase the noise level slowly during the train 2) Try with different subparts of the sphere (order 2 ->48 and 1-> 12) The reader expects a plot with order 1 or 2

[nperraud]

Result for order=2

Augmenting the dataset using different random noise is just making the difference.

Increasing slowly the noise level seems to make the optimization more stable but it might also be the fact that the step size is decreased with time. It does not seem very needed.

Noise	Histogram	SCNN	Epochs
0	0%	0%	1
5	0%	0%	5
10	1.43%	0.95%	10	Peak 0.48%
15	21.67%	15%	10
20	40.71%	23.57%	20

I will run the simulations for order 1 and 4 soon.

[nperraud]

Result for order=4

The sphere is cut into 192 parts.

Noise	Histogram	SCNN	Epochs
0	0%	0%	1
5	0.18%	0%	2
10	14.4%	4.17%	10
15	36.73%	23.15%	10
20	46.13%	33.81%	10

[nperraud]

Result for order=1

The sphere is cut into 12 parts.

Noise	Histogram	SCNN	Epochs	Comments
0	0%	0%	5
10	0%	0%	10
15	7.22%	2.78%	40	Peak 1.11%
20	37.22%	18.89%	40	Peak 16.67%

[mdeff]

These are very nice results! :) Why is the problem harder with a smaller sub-division of the sphere though?

[nperraud]

You get less data, so the histogram is less precise.

[mdeff]

Make sense :)

[mdeff]

I actually wonder if our comparison is valid. The problem is that the graph CNN sees way more examples (due to data augmentation) than the SVM which only sees one instance of the noise. For it to be fair the same training data should be used for both models.

[nperraud]

I actually thought about this. I did not really do extensive tests because (a) it takes time and (b) I observed for the order 4, that adding more samples was not changing the accuracy of the histogram classification. I am not entirely sure, it is the same for order 2 and 1. Do you think, we should test it to be sure?

[mdeff]

I see. But I really think we should use the same (augmented) training data for a fair comparison. Especially as the robustness to noise with data augmentation is the selling point of our method.

[nperraud]

Actually, I was thinking to simply do a plot with the number of samples on axis 'x' and the acurracy on axis 'y'. I can make a notebook just for that and the we run it for the different orders and noise levels.

[mdeff]

That sounds like a very good idea. :+1: Hopefully the histogram method will reach a plateau while ours will keep becoming better.

[nperraud]

All experiments done. Check the results in the paper.