This PR adds several small tweaks/enhancements. Most (though not all) are one commit each, so it might be easier to look at the commits separately rather than the overall diff. We could also cherry-pick the ones you like, or I could make 14 PRs :-)
Paper dataset generation configs:
Rename Configs/2203_00690 to Configs/paper_2203_00690 (subpackage names can't start with a number), add an __init__.py to the folder, and replace xxxx_yyyy in the files with the actual folder name. This way the configurations work, and they can be imported.
Remove parameter specifications in the config that are overridden by cross_dict, preventing a warning
Add leading zeros to test set config filenames, so sorted(glob('config_test_*.py')) returns the test configs in order.
Do copy.deepcopy on the config_dict before mutating it in derived configs. Otherwise, if you import multiple configs in one python session, the config modules all mutate a single dictionary, so the user doesn't actually get separate configs..
Image generation: (all to avoid always-on warnings)
Disable KWARGSNUMERICWARNING1/KWARGSNUMERICWARNING2 by default. These warnings are always thrown the first time a drizzled image is generated, they don't indicate a problem, and there is nothing the user can/should do about them.
Avoid using the deprecated pd.DataFrame.append function when preparing metadata to csv, and remove some duplication in the metadata writing code.
List some columns to be left out of metadata by default, which prevents the 'column could not be serialized' warning to be always thrown. Now, if you see the warning, something new you added isn't getting written.
Tensorflow dataset generation:
tfrecord image order now matches sorted glob of image files
shuffle=False option, useful for re-using the dataset loading code for test data
Inference:
When checking if the covariance matrix is positive-definite, check symmetry and existence of Cholesky decomposition rather than doing an eigendecomposition and checking the eigenvalues are positive. Roughly doubles the inference speed for me; see https://stackoverflow.com/questions/16266720 for discussion, you'll want to read several of the answers.
Catch-and-return-inf on a few more singular cases. May only happen if MCMC is being very naughty.
Tests:
add an ini so people can attempt to use pytest as the test runner. Unfortunately, I still get some mysterious numba errors when using pytest that don't appear when using unittest. That might be specific to my setup/numba version though, not sure.
Use a batch size of 8, not 1000, to test xresnet34. My GPU doesn't have enough RAM for 1000.
Misc:
Backport #34 (including 64->32 fix) to main. Should help you get a clean diff when you compare main and redshift_input, and helps me get the benefits of that PR when making new test sets off main.
Add the cosmos dataset directory to gitignore. Prevents git from slowing down / complaining about dirty working directory during some operations.
This PR adds several small tweaks/enhancements. Most (though not all) are one commit each, so it might be easier to look at the commits separately rather than the overall diff. We could also cherry-pick the ones you like, or I could make 14 PRs :-)
Paper dataset generation configs:
Configs/2203_00690
toConfigs/paper_2203_00690
(subpackage names can't start with a number), add an__init__.py
to the folder, and replacexxxx_yyyy
in the files with the actual folder name. This way the configurations work, and they can be imported.sorted(glob('config_test_*.py'))
returns the test configs in order.copy.deepcopy
on theconfig_dict
before mutating it in derived configs. Otherwise, if you import multiple configs in one python session, the config modules all mutate a single dictionary, so the user doesn't actually get separate configs..Image generation: (all to avoid always-on warnings)
pd.DataFrame.append
function when preparing metadata to csv, and remove some duplication in the metadata writing code.Tensorflow dataset generation:
shuffle=False
option, useful for re-using the dataset loading code for test dataInference:
Tests:
Misc: