Closed SteveOv closed 4 weeks ago
As an example, with $k=0.2$, we can derive the following values for qphot
Current model: $q_{phot} = k^{2.0} = 0.04$ (then select from a normal dist with this as the origin)
Both low mass: $q{phot} = k^{2.5} \approx 0.018$ Both high mass: $q{phot} = k^{1.25} \approx 0.13$
However, if the actual radii were 1.5 and 7.5 Rsun the appropriate low/high mass relation would give us; $$q{phot} = \frac{R_B^{2.5}}{R_A^{1.25}} = \frac{1.5^{2.5}}{7.5^{1.25}} \approx \frac{2.76}{12.4} \approx 0.22$$
Some options;
Have added qphot to the superset of possible labels serialized to the dataset files (tfrecord) with #72
Just checked; existing control fits are unchanged by this as qphot label is ignored unless it's in the estimator's label_names (had to be sure)
If we include the qphot label values in the control fits (fit flag == 0, so not fitted) we get an overall improvement on the fitted params with MAE for the current 23 systems going from 0.006596 to 0.005826. All systems, except the following show improvements;
Nothing completely broken.
If I want to predict qphot I'll need to address the following;
did a trial run of predicting qphot (with qphot = normal(loc=k**2, scale=0.1)) and the results were poor
Metrics for 20000 system(s).
---------------------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J qphot ecosw esinw bP MAE MSE
===============================================================================================================
MAE | 0.007465 0.076421 0.092982 0.212754 0.028889 0.039254 0.081013 0.076968
MSE | 0.000148 0.015114 0.079022 0.132493 0.005630 0.005313 0.016215 0.036276
Currently building a training set with qphot fixed at -100.
Now training a model ... Oh dear
Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.011885 0.171784 0.119085 0.035280 0.044967 0.183468 0.094412
MSE | 0.000385 0.056839 0.096361 0.007147 0.007519 0.061416 0.038278
compared with
Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.006927 0.069053 0.076646 0.023456 0.034658 0.065675 0.046069
MSE | 0.000140 0.013702 0.066056 0.004221 0.004675 0.012697 0.016915
Some empirical relations fairly widely cited with Demircan & Kahraman (1991) k-q relations being particularly useful (pp 320);
k = q^{0.935} \text{ where } M_A \text{ and } M_B < 1.66 \text{M}_{\odot}
k = q^{0.542} \text{ where } M_A \text{ and } M_B > 1.66 \text{M}_{\odot}
k = q^{0.724} \text{ where } M_A > 1.66 \text{M}_{\odot} \text{ and } M_B < 1.66 \text{M}_{\odot}
These could be directly applicable as $q \simeq k^{1.07}$, $q \simeq k^{1.85}$ and $q \simeq k^{1.38}$ respectively.
Having tested models trained of various combinations of the Demircan & Kahraman k-q relations, I've found the best result is to use their single R-M relation;
R \simeq 1.01 M^{0.724}, \text{ where } 0.1 \text{M}_{\odot} < M < 18.1 \text{M}_{\odot}
giving $q_{phot} \simeq k^{1.4}$. By applying a normal distribution to the exponent the training data has sufficient variation to cover the range of the relations in the previous comment, and gives a Model that performs the best on the test dataset and real systems.
qphot = k**np.random.normal(1.4, scale=0.2)
Synthetic test dataset.
Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.008465 0.070938 0.084118 0.030409 0.036350 0.071506 0.050298
MSE | 0.000256 0.013899 0.072194 0.005516 0.005028 0.014322 0.018536
Real systems.
Metrics for 23 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.011154 0.107377 0.069382 0.004972 0.026247 0.107491 0.054437
MSE | 0.000420 0.025136 0.010503 0.000038 0.001674 0.019837 0.009601
The results om the real systems are consistent across transiting and non-transiting subsets, with MAE of 0.054 and 0.055 respectively. I've found most models predict the transiting subset signifantly less well.
NOTE: these results and those that follow are not directly comperable with any in previous comment, as there has been a change to the allowable range for rA and rB which has affected reported stats. The revised baseline is
Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.007319 0.072676 0.085889 0.025034 0.035625 0.071202 0.049624
MSE | 0.000157 0.014805 0.071523 0.004674 0.004405 0.013543 0.018184
The best of the alternative distributions tried;
if k < 0:
# Invalid and will cause errors in the qphot calculation
# This will be discarded as an unusable instance
qphot = k
elif k <= 0.8:
# Secondary significantly smaller, which brings the mixed mass regime into play
qphot = k**np.random.choice([1.07, 1.85, 1.38], p=[0.50, 0.25, 0.25])
else:
qphot = k**np.random.choice([1.07, 1.85], p=[0.50, 0.50])
Synthetic test dataset
Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.007538 0.079808 0.088545 0.023032 0.041081 0.076420 0.052737
MSE | 0.000202 0.017118 0.077915 0.004277 0.005371 0.015840 0.020120
Real systems. While the headline MAE for the real systems is slightly better than the chosen model, the predictions for the transiting subset are significantly worse with an MAE of 0.068.
Metrics for 23 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.013762 0.104215 0.069650 0.004567 0.032602 0.091275 0.052679
MSE | 0.000424 0.021485 0.012381 0.000048 0.002303 0.020924 0.009594
Did some major testing of various distributions of k, J and qphot with reduced trainsets of 100k instances. The best compromise was to keep the current k & J distribution and revise the qphot distribution to use;
qphot = np.random.normal(loc=k**1.4, scale=0.3) if k > 0 else 0
yielding the following result for the synthetic test dataset
Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.007074 0.073505 0.082858 0.023896 0.035243 0.072011 0.049098
MSE | 0.000115 0.014089 0.068914 0.004474 0.004856 0.013523 0.017662
and this for the formal test set (before swapping out V454 Aur for V456 Cyg):
Metrics for 23 system(s).
----------------------------------------------------------------------------------------------------
Summary | rA_plus_rB k J ecosw esinw bP MAE MSE
====================================================================================================
MAE | 0.014499 0.112811 0.048683 0.005223 0.026908 0.090902 0.049838
MSE | 0.000514 0.023675 0.006383 0.000050 0.001427 0.016308 0.008060
Note: these results were produced from a model trained on 80k/20k train/validation instances so the results are not directly comperable to those above which were trained on 200k/50k instance.
One of the params we provide to JKTEBOP task 2 when generating training instances is qphot, the mass ratio used to model mass related stellar distortion.
We don't have any absolute params for the instance, so the current way of calculating this is based on the homology M-R relations applied to k (ratio of radii);
This needs revising to something more sophisticated. A slight improvement is to randomly choose either the low or high mass relation;
however neither approach models a low mass secondary/high mass primary at all well.