This PR includes a Github Actions workflow defined in ci.yml, which runs some unit tests for some methods in io.py module.
There are also two validation tests for train.py and metrics.py, respectively. The baseline used for validation are the outputs of the main branch run over two cycles and epochs with a specific random seed used for initialisation of tensor flow. These can't be run as unit tests because they require GPUs which are scarcely available when launching the CI.
To validate the training results, a mock run of the refactored code is launched using the same configuration as the main branch. Results are stored in the pytest fixture tmp_path_factory. The checkpoint weights and final psf models are read in and compared with the baseline results and rms is computed. The validation test asserts for each weight that the error < 1.e-9.
To validate the metrics, there are four tests to evaluate the metrics: low-res pixel reconstruction metric, optical path difference metric, monochromatic metric, and shape metrics produced by the refactored code with respect to the metrics computed by the main branch again ran over exactly the same configuration. The weights are read in and the psf model is regenerated and evaluated according to each metric. The tolerance level varies in this case where low res the tol is set to 1e-7 whereas the rest is 1e.-9. Bc I saw that the difference was ~4.8e-8 for the low-res pixel reconstruction.
I merged changes in the last PR to this branch. I will work on the end-to-end test in another branch.
This PR includes a Github Actions workflow defined in
ci.yml, which runs some unit tests for some methods inio.pymodule.There are also two validation tests for train.py and metrics.py, respectively. The baseline used for validation are the outputs of the main branch run over two cycles and epochs with a specific random seed used for initialisation of tensor flow. These can't be run as unit tests because they require GPUs which are scarcely available when launching the CI.
To validate the training results, a mock run of the refactored code is launched using the same configuration as the main branch. Results are stored in the pytest fixture tmp_path_factory. The checkpoint weights and final psf models are read in and compared with the baseline results and rms is computed. The validation test asserts for each weight that the error < 1.e-9.
To validate the metrics, there are four tests to evaluate the metrics: low-res pixel reconstruction metric, optical path difference metric, monochromatic metric, and shape metrics produced by the refactored code with respect to the metrics computed by the main branch again ran over exactly the same configuration. The weights are read in and the psf model is regenerated and evaluated according to each metric. The tolerance level varies in this case where low res the tol is set to 1e-7 whereas the rest is 1e.-9. Bc I saw that the difference was ~4.8e-8 for the low-res pixel reconstruction.
I merged changes in the last PR to this branch. I will work on the end-to-end test in another branch.
Unused modules were removed for the release.
This PR resolves resolves issue https://github.com/CosmoStat/wf-psf/issues/47