better handling of qphot

[SteveOv]

One of the params we provide to JKTEBOP task 2 when generating training instances is qphot, the mass ratio used to model mass related stellar distortion.

We don't have any absolute params for the instance, so the current way of calculating this is based on the homology M-R relations applied to k (ratio of radii);

            # The qphot mass ratio value (MB/MA) affects the lightcurve via the ellipsoidal effect
            # from the distortion of the stars' shape. Generate a value from the ratio of the radii
            # (or set to -1 to force spherical). Standard homology M-R ratios are a starting point;
            # - low mass M-S stars;     M \propto R^2.5
            # - high-mass M-S stars;    M \propto R^1.25
            qphot       = np.random.normal(loc=k**2, scale=0.1)

This needs revising to something more sophisticated. A slight improvement is to randomly choose either the low or high mass relation;

            qphot       = np.random.normal(loc=np.random.choice([k**2.5, k**1.25], p=[0.75, 0.25]),
                                           scale=0.1)

however neither approach models a low mass secondary/high mass primary at all well.

[SteveOv]

As an example, with $k=0.2$, we can derive the following values for qphot

Current model: $q_{phot} = k^{2.0} = 0.04$ (then select from a normal dist with this as the origin)

Both low mass: $q{phot} = k^{2.5} \approx 0.018$ Both high mass: $q{phot} = k^{1.25} \approx 0.13$

However, if the actual radii were 1.5 and 7.5 Rsun the appropriate low/high mass relation would give us; $$q{phot} = \frac{R_B^{2.5}}{R_A^{1.25}} = \frac{1.5^{2.5}}{7.5^{1.25}} \approx \frac{2.76}{12.4} \approx 0.22$$

[SteveOv]

Some options;

• can I use a relation based on $J$?
• just set qphot to -100 always to force spherical (or perhaps when rA_plus_rB > some threshold)
• use a distribution qphot values and train ebop_maven to predict
  • much like with k and J, there's no need to worry about plausibility; we're looking to train the model to derive a relation implicit in the variable's impact on light-curve morphology

[SteveOv]

Have added qphot to the superset of possible labels serialized to the dataset files (tfrecord) with #72

Just checked; existing control fits are unchanged by this as qphot label is ignored unless it's in the estimator's label_names (had to be sure)

If we include the qphot label values in the control fits (fit flag == 0, so not fitted) we get an overall improvement on the fitted params with MAE for the current 23 systems going from 0.006596 to 0.005826. All systems, except the following show improvements;

• V539 Ara; MAE from 0.008 to 0.014
• V570 Per; MAE from 0.007 to 0.010
• ZZ Uma; MAE from 0.0038 to 0.0066

Nothing completely broken.

[SteveOv]

If I want to predict qphot I'll need to address the following;

• what distribution to use; normal like for k and J?
• make qphot fittable in the task3.in.template
  • most flexible approach is to give it a ${qphot_fit} token and default this to 1 in model_testing.base_jktebop_task3_params()
  • potentially set qphot_fit to 0 for fit_overrides in formal-test-dataset.json where qphot specified (psi Cen, V362 Pav, V530 Ori)
• model_testing
  • fit_against_formal_test_dataset(); add qphot to fit_names
• model_interactive_tester:
  • add qphot to fit_prms (used to read fitted params from jktebop par file)
• the general issue of what to do when there are fit overrides for this
  • the overrides are taken from published works and are used to secure a fit consistent with that published and as such they are essential to the control fit
  • so setting qphot_fit to zero (which is effectively what we have now as this flag is 0 in the template) seems a "no brainer" for the control fit
  • but this isn't always appropriate - when using the estimator, other estimated params are all left free to fit

did a trial run of predicting qphot (with qphot = normal(loc=k**2, scale=0.1)) and the results were poor

• qphot_fit = 1 by default (unless overridden in fit_overrides)
• the control fits were affected by the qphot label (fitted) which lead to significant deviation from expected results
• predictions for qphot were siginificantly worse than other labels
  • for synth test the MAE(qphot) was 0.213
  • for formal test the MAE(qphot) was 0.251
  • more than double the MAE for any other predicted value
  • regardless of whether the function for qphot is good, or not, the model was poor at learning it

Metrics for 20000 system(s).
---------------------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      qphot      ecosw      esinw         bP        MAE        MSE
===============================================================================================================
MAE        |   0.007465   0.076421   0.092982   0.212754   0.028889   0.039254   0.081013   0.076968
MSE        |   0.000148   0.015114   0.079022   0.132493   0.005630   0.005313   0.016215              0.036276

[SteveOv]

Currently building a training set with qphot fixed at -100.

Now training a model ... Oh dear

Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.011885   0.171784   0.119085   0.035280   0.044967   0.183468   0.094412
MSE        |   0.000385   0.056839   0.096361   0.007147   0.007519   0.061416              0.038278

compared with

Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.006927   0.069053   0.076646   0.023456   0.034658   0.065675   0.046069
MSE        |   0.000140   0.013702   0.066056   0.004221   0.004675   0.012697              0.016915

[SteveOv]

Some empirical relations fairly widely cited with Demircan & Kahraman (1991) k-q relations being particularly useful (pp 320);

k = q^{0.935} \text{ where } M_A \text{ and } M_B < 1.66 \text{M}_{\odot}

k = q^{0.542} \text{ where } M_A \text{ and } M_B > 1.66 \text{M}_{\odot}

k = q^{0.724} \text{ where } M_A > 1.66 \text{M}_{\odot} \text{ and } M_B < 1.66 \text{M}_{\odot}

These could be directly applicable as $q \simeq k^{1.07}$, $q \simeq k^{1.85}$ and $q \simeq k^{1.38}$ respectively.

[SteveOv]

Having tested models trained of various combinations of the Demircan & Kahraman k-q relations, I've found the best result is to use their single R-M relation;

R \simeq 1.01 M^{0.724}, \text{ where } 0.1 \text{M}_{\odot} < M < 18.1 \text{M}_{\odot}

giving $q_{phot} \simeq k^{1.4}$. By applying a normal distribution to the exponent the training data has sufficient variation to cover the range of the relations in the previous comment, and gives a Model that performs the best on the test dataset and real systems.

            qphot       = k**np.random.normal(1.4, scale=0.2)

Synthetic test dataset.

Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.008465   0.070938   0.084118   0.030409   0.036350   0.071506   0.050298
MSE        |   0.000256   0.013899   0.072194   0.005516   0.005028   0.014322              0.018536

Real systems.

Metrics for 23 system(s). 
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.011154   0.107377   0.069382   0.004972   0.026247   0.107491   0.054437
MSE        |   0.000420   0.025136   0.010503   0.000038   0.001674   0.019837              0.009601

The results om the real systems are consistent across transiting and non-transiting subsets, with MAE of 0.054 and 0.055 respectively. I've found most models predict the transiting subset signifantly less well.

NOTE: these results and those that follow are not directly comperable with any in previous comment, as there has been a change to the allowable range for rA and rB which has affected reported stats. The revised baseline is

Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.007319   0.072676   0.085889   0.025034   0.035625   0.071202   0.049624
MSE        |   0.000157   0.014805   0.071523   0.004674   0.004405   0.013543              0.018184

[SteveOv]

The best of the alternative distributions tried;

            if k < 0:
                # Invalid and will cause errors in the qphot calculation
                # This will be discarded as an unusable instance
                qphot = k
            elif k <= 0.8:
                # Secondary significantly smaller, which brings the mixed mass regime into play
                qphot = k**np.random.choice([1.07, 1.85, 1.38], p=[0.50, 0.25, 0.25])
            else:
                qphot = k**np.random.choice([1.07, 1.85], p=[0.50, 0.50])

Synthetic test dataset

Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.007538   0.079808   0.088545   0.023032   0.041081   0.076420   0.052737
MSE        |   0.000202   0.017118   0.077915   0.004277   0.005371   0.015840              0.020120

Real systems. While the headline MAE for the real systems is slightly better than the chosen model, the predictions for the transiting subset are significantly worse with an MAE of 0.068.

Metrics for 23 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.013762   0.104215   0.069650   0.004567   0.032602   0.091275   0.052679
MSE        |   0.000424   0.021485   0.012381   0.000048   0.002303   0.020924              0.009594

[SteveOv]

Did some major testing of various distributions of k, J and qphot with reduced trainsets of 100k instances. The best compromise was to keep the current k & J distribution and revise the qphot distribution to use;

qphot       = np.random.normal(loc=k**1.4, scale=0.3) if k > 0 else 0

yielding the following result for the synthetic test dataset

Metrics for 20000 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.007074   0.073505   0.082858   0.023896   0.035243   0.072011   0.049098
MSE        |   0.000115   0.014089   0.068914   0.004474   0.004856   0.013523              0.017662

and this for the formal test set (before swapping out V454 Aur for V456 Cyg):

Metrics for 23 system(s).
----------------------------------------------------------------------------------------------------
Summary    | rA_plus_rB          k          J      ecosw      esinw         bP        MAE        MSE
====================================================================================================
MAE        |   0.014499   0.112811   0.048683   0.005223   0.026908   0.090902   0.049838
MSE        |   0.000514   0.023675   0.006383   0.000050   0.001427   0.016308              0.008060

Note: these results were produced from a model trained on 80k/20k train/validation instances so the results are not directly comperable to those above which were trained on 200k/50k instance.