ratt-ru / QuartiCal

CubiCal, but with greater power.
MIT License
8 stars 4 forks source link

Adding robust weights #86

Closed JSKenyon closed 2 years ago

JSKenyon commented 3 years ago

@ulricharmel I have added a solver.robust option which gets piped all the way down to the solvers. You can pull it out from the meta_args. I have also added the ability to write out the weights to a column of your choosing. This is done as part of the normal output arguments e.g. output.products=[corrected_residuals,weights] and output.columns=[CORRECTED_DATA,ROBUST_WEIGHT] will write the corrected residuals to the CORRECTED_DATA column and the weights to a new ROBUST_WEIGHT column (although you can write back to WEIGHT_SPECTRUM if you want).

I have not done any work inside the kernels. You will have to figure that out, although I am happy to help. It is very important that you try to write the robust reweighting in such a way that it mutates the contents of the weight variable in place. This will probably be the trickiest part (making sure zero weights remain zero), but I will definitely try and help.

Feel free to branch off this branch, or work on it directly.

JSKenyon commented 3 years ago

Note that this PR is now an alternative to #93. The reason for creating a new PR rather than using @ulricharmel's directly is so that I can develop a deeper understanding of the robust reweighting and integrate in a slightly different way.

All credit for the robust reweighting goes to @ulricharmel.

JSKenyon commented 3 years ago

@bennahugo @landmanbester This branch will also include the option to output corrected weights (equivalent to calibration weights) and to specify an input sigma column from which the weights will be reinitialized.

JSKenyon commented 2 years ago

I am going to merge this - I am relatively confident that it is in a working (though perhaps not yet perfect) state, and that it does nothing unless enabled. We can now experiment to see what additional features would be useful e.g. reweighting based on baseline length etc.

ulricharmel commented 2 years ago

Hi @JSKenyon . I was just going through the code and everything is smooth. I am just setting a parset now to run it on the DEEP2 dataset to see if we get the same results as before.