search

LSSTDESC / tomo_challenge

2020 Tomographic binning challenge
13 stars 18 forks source link

Unexpected trend in SNR_ww metric #28

Open belaa opened 4 years ago

belaa commented 4 years ago

@theabbybault and I have been investigating the effect of number of bins on SNR metrics using the random forest classifier. We have found that the ww scores are showing only marginal changes as a function of increasing bins, and it seems like the shear information isn't affecting the value of SNR_3x2 at all. The only thing we've changed in random_forest.py is the addition of a random seed in line 81. We've tried running with different seeds and still recover the same trend. Here is a link to the notebook we are using to call the random forest classifier (using seed=123), including a plot of the results at the very bottom (also shown below). Screen Shot 2020-08-05 at 10 55 02 AM

dylanagreen commented 4 years ago

If I can add to this, I have been experimenting with my own personal binning method that is not based on the random forest, and it appears to show the same trend. This should assuage any questions about it being a random forest specific trend. hist(1)

pwhatfield commented 4 years ago

A minor addition to this, I have been doing similar simple experiments with number of bins, finding that the Fisher FOM values occasionally come out anomalously high/doesn't seem to have a smooth dependence on bin number, is this to be expected/anyone else had this problem or do I likely have it set up incorrectly?

tomo_1 tomo_2 tomo_4