HinLeung622
commented
4 years ago @grd349
But I have more results: NGC6791, a 8Gyr cluster that is metal rich (feh = 0.4) and helium rich (Y=0.3):
Location on our training data:
It only has 31 stars, after we removed the lower RGB.
Results:
Super good Rhats, and that is with only 5000 tuning steps, 1/4 the amount needed for M67 above.
Great agreement between chains, the only problem seems to be mixing length, that is all the way at the edge of our training data range (1.7-2.3). Corner plot displays all the same correlations mentioned above (but to a smaller extent).
Sampling regions:
HBM is able to recover the correct stars in the correct locations on the HR.
But problem shows up when we plot the fitted isochrone:
That isochrone, skipping the dip in the subgaint, is just wrong, and this is a neural net problem. This links back to how the NN we are currently using, is not doing well in the feh dimension:
These plots shows how well the NN is able to generalize the behaviour of the data in between the tracks, in mass, feh, Y and MLT space. The plots look a little mess, due to I deliberately shifting the tracks in the direction that will speed up stellar evolution, causing the excess at the RGB that sometimes bend back. But anyhow, it is very clear that in the mass, Y and MLT dimensions, the NN is doing a much better job in generalizing in between tracks within the boundaries of training data, than in feh space. I have talked to Alex about this, and he suggested to try taking out batch normalization, as he has seen certain patterns in our feh varying plot before in his batch norm experiments. So I have trained up a NN that does not use any batch norm, and am now running a HBM with that NN. It seems to have shown slightly better generalizability in the feh space:
Again, I will report back when that HBM is finished.
On the other hand, seeing the HBM results for NGC6791, Harry suggested the possible solution of training cluster-specific NNs, as we speculated one of the reasons NGC6791's isochrone is looking less good compared to M67, is due to its metallicity and helium fraction being more distant from solar values than M67 is, which we know that NN does best at its central, solar initial conditions. So Harry is experimenting with training a NN only one NGC6791-relevant data, but is getting possibly better results on trying to improve the feh generalizability.
Ignoring the top feh track that is outside the training sample, the other in between feh tracks are well within the boundaries of the black, on track lines, which is a good sign.
Another problem we have spotted (I have knew this for a week now but it didn't impact our HBM so I not mention it), is overfitting in the obversable-age dimension:
This is a observable vs age plot of one random track. We are plotting 10 additional NN predict points between each grid data point. And the spread of blue dots caused by the "in between" dots is clearly a sign of overfitting. We have tried increasing regularization to combat this problem, but it seems the NN's ability to produce good predictions in all input dimensions, and the loss we are able to push it down to, is quite sensitive to regularization, so we are kind of stuck. Harry will be looking further into this before Monday.
grd349
And all the stars are pretty off.
This HBM is done with the same NN as the one in NGC188, and the deviate from subgiant of the isochrone is caused by overfitting of the NN in feh space, which we know is a persistent problem in our final NNs. This problem gets worse the more the cluster deviates from solar metallicities, and NGC6791 here is +0.4 Fe/H. NGC188 is estimated to have an feh of 0.08, which isn't much from solar values, but that might be causing the problem together with the lack of MS stars data.
This is the reason why we had to train a NN on NGC6791-specific training data in hopes that there will be less overfitting in feh space, which gives the HBM result in NGC6791 trial 5:
What do you think about this? To me it seems like even at the MS, the stars' luminosities are just a little bit too low. If they were raised by say, 0.1 in log luminosity, the isochrone will get a much better fit in the MS. We already went back and checked our luminosity calculations for the cluster the day we altered the stars we put into this cluster, and there didn't seem to be any problems, so I am not sure how to improve on this and get the HBM to estimate properly.
@grd349 And after a few months of hard work a lot of problems that we had to solve (or give up on), today I can finally present some of the first "report level" results:
First of all, after my previous issue post, Harry pointed out a very big detail that I overlooked when we selected our data for our clusters: I was treating the cluster data like our training data, and included some low RGB stars in the cluster data. Given that we already know our RGB will be doing badly, why even bother to leave in those RGB stars at the first place? So with this, we removed some more RGB stars, from this:
to this:
I put on an (basically) identical HBM run as last time for M67, but for way more tuning steps, and here is the results:
pretty good Rhats for most of the hyperpriors, feh seems to be the most problematic.
Great agreement between the chains on everything outside feh. Mean helium is off to one side, this is due to the prior being Beta(1.1,1.1)*0.3+0.26, which the estimated mean helium of M67 seems to be closer to 0.26 than I first expected (as I went off the assumption that M67 is basically solar initial conditions).
But even with helium and feh being slightly problematic, the correlations reported in McKeever's paper on NGC6791, and other known correlations in clusters is clearly visible: the anti-correlation between mean_Y and mean_age, between mean_feh and mean_age, and the positive correlation between mean_Y and mean_feh. Interestingly, there is a negative correlation between mean_Y and mean_MLT, don't know why this is the case.
For sampling regions:
Pretty good.
I also figured out how to plot the isochrone along with a 1sigma spread around it under the cluster's data:
Dark grey is where stars would lie if they were within 1x spread value in age, feh and/or helium away from the mean cluster values. Light grey is 2x spread. I cut off the shaded regions plots at early RGB due to the shaded regions are done by NN predictions, and the predictions fall apart at the RGB, creating very ugly and wrong shapes.
Note that this HBM result uses a version of M67 cluster data that still has its low RGB stars not-removed (as discussed above). I am currently running another HBM that has those stars removed and also with a wider and lower mean_Y hyperprior. I will report the results when I have them.