This adds support for end-to-end robustness tests.
A new script, robustness_test.py, allows you to quickly do the main sequence of tasks -- generate images, get predictions from a network, and run a hierarchical inference MCMC. You can run the config as-is, or specify which parts of the config to change -- e.g. set the population's sigma_sub or theta_E to a different distribution or a fixed value.
The new paltas.Analysis.gaussian_inference file supports this with:
A run_network_on function that runs a neural network over all images in a folder, rotationally averaging predictions similar to the paltas paper. The main difference is that this code rotates over a deterministic set of rotations (e.g. 0 degrees, 10 degrees, 20, etc) rather than a random set -- for n_rotations large enough this should not matter. Results are saved in an npz in the image folder, or returned.
A GaussianInference class to help coordinate hierarchical inference. You can initialize it with a folder you previously did run_network_on on (with .from_folder), or just the npz it produces (with .from_npz). GaussianInference instances have a .bayesian_mcmc method that runs hierarchical inference as described in the paper. Only difference is that the hyperpriors on std_[population_parameter] are uniform in [0, inf] rather than uniform in some range of log, to avoid concentrating the hyperprior at very small values. You can revert to the log prior by setting log_sigma=True when building GaussianInference.
Feel free to tell me some or all of this doesn't belong in paltas! This is probably an alternative implementation of the stuff you already have in jupyter notebooks; would be useful to compare them at some point.
Other changes
generate.py and robustness_test.py can now both be called through functions as well as shell commands. A small command line interface generator (in Utils.cli_maker) takes care of this. This also simplifies some test code.
Include .cvs and .fits files already in the repo in the python package. On the main branch, setuptools omits bad_galaxies.csv etc from the wheel, since it's not listed as package data. As long as you do a developer/editable install this doesn't matter, but if you do a regular pip install this will bite you (at least when using the paper configs). I had to move the HST PSFs to somewhere under paltas (and add an __init__.py to the folder); I don't think it's easy to control where data files that live outside the package get shipped. A directory symlink ensures that any old code that relies on this location still works.
Sampler changes:
921355716309c2b0b2492d73f40482d2255aef69: Allow specifying fixed values for parameters that expect multiple values
2b3950dab6131d43853172152626e1a42a7e021b: Temporary workaround for https://github.com/scipy/scipy/issues/16998. This ensures they scipy.stats distributions use the common random state before we draw from them. Without this, setting the random seed only has full effect on the training config, since other configs deepcopy the training config's config_dict. This is a hack, and for now only supports plain scipy.stats distributions and Duplicate distribution, not the other custom distributions paltas implements.
f267267819f5a291975e6edd134891c73b800d25: Feel free to disagree with this, happy to revert. This demotes some global variables in Analysis/hierarchical_inference.py to class attributes. The globals existed to allow the use of multiprocess.Pool in the MCMC inference. I think:
Most inferences will run in single-core batch jobs, for which multiprocessing isn't available anyway.
Global variables are :imp:... or at least these particular ones caused confusion for me when I started working on this PR, and I can imagine they might hurt future users' brains too.
For running locally, you generally want to run multiple configurations. This can still be parallelized, e.g.
from concurrent.futures import ProcessPoolExecutor, as_completed
with ProcessPoolExecutor(max_workers=6) as exc:
as_completed(exc.map(do_inference, folders))
* f267267819f5a291975e6edd134891c73b800d25, f267267819f5a291975e6edd134891c73b800d25: `unnormalize` can now also unnormalize the precision matrix directly, and can be run independently for unnormalizing means and covariances
* d28e29a9f1342f7abc7e227633752785e141c66f, a4774f150ee48abd51f2cf81f3f239ed9a7eeeaa: add options to artificially make source galaxy images larger or brighter (or smaller/fainter).
Coverage decreased (-1.09%) to 93.983% when pulling 9db46e1832dae2391972d7b75cdd12015250db8c on gaussian_inference_helper into 112a4a5b1eadbeeac28b9b251a9a411c1d050161 on main.
I moved most of this to a separate repository here https://github.com/JelleAalbers/bendorbreak, so we can close this. I might make a new PR for some of the smaller fixes in here.
This adds support for end-to-end robustness tests.
A new script,
robustness_test.py
, allows you to quickly do the main sequence of tasks -- generate images, get predictions from a network, and run a hierarchical inference MCMC. You can run the config as-is, or specify which parts of the config to change -- e.g. set the population's sigma_sub or theta_E to a different distribution or a fixed value.The new
paltas.Analysis.gaussian_inference
file supports this with:run_network_on
function that runs a neural network over all images in a folder, rotationally averaging predictions similar to the paltas paper. The main difference is that this code rotates over a deterministic set of rotations (e.g. 0 degrees, 10 degrees, 20, etc) rather than a random set -- forn_rotations
large enough this should not matter. Results are saved in an npz in the image folder, or returned.GaussianInference
class to help coordinate hierarchical inference. You can initialize it with a folder you previously didrun_network_on
on (with.from_folder
), or just the npz it produces (with.from_npz
).GaussianInference
instances have a.bayesian_mcmc
method that runs hierarchical inference as described in the paper. Only difference is that the hyperpriors on std_[population_parameter] are uniform in [0, inf] rather than uniform in some range of log, to avoid concentrating the hyperprior at very small values. You can revert to the log prior by settinglog_sigma=True
when buildingGaussianInference
.Feel free to tell me some or all of this doesn't belong in paltas! This is probably an alternative implementation of the stuff you already have in jupyter notebooks; would be useful to compare them at some point.
Other changes
generate.py
androbustness_test.py
can now both be called through functions as well as shell commands. A small command line interface generator (inUtils.cli_maker
) takes care of this. This also simplifies some test code..cvs
and.fits
files already in the repo in the python package. On the main branch, setuptools omitsbad_galaxies.csv
etc from the wheel, since it's not listed as package data. As long as you do a developer/editable install this doesn't matter, but if you do a regular pip install this will bite you (at least when using the paper configs). I had to move the HST PSFs to somewhere underpaltas
(and add an__init__.py
to the folder); I don't think it's easy to control where data files that live outside the package get shipped. A directory symlink ensures that any old code that relies on this location still works.deepcopy
the training config'sconfig_dict
. This is a hack, and for now only supports plain scipy.stats distributions andDuplicate
distribution, not the other custom distributions paltas implements.Analysis/hierarchical_inference.py
to class attributes. The globals existed to allow the use ofmultiprocess.Pool
in the MCMC inference. I think:def do_inference(folder): inf = gi.GaussianInference.from_folder(folder) summary, chain = inf.bayesian_mcmc()
save results somewhere
with ProcessPoolExecutor(max_workers=6) as exc: as_completed(exc.map(do_inference, folders))