search

jluastro / PopSyCLE

Population Synthesis for Compact-object Lensing Events
https://popsycle.readthedocs.io/en/latest/
16 stars 3 forks source link

Adding ZTF filters and future support for additional filters #16

Closed MichaelMedford closed 4 years ago

MichaelMedford commented 4 years ago

This series of commits adds the abilities to calculate ZTF (Zwicky Transient Facility) filters from UBV filters, and then place that photometry into all data products of the pipeline (except the original ebf file which remains untouched). Here is an outline for the additional functions and changes to existing functions in this branch:

New arguments: photometric_system and additional_photometric_systems Supporting additional filters requires allowing the user to select adding additional photometry to PopSyCLE's data products. This is achieved in two ways in two different places.

In perform_pop_syn and calc_events, the user can provide a list of additional_photometric_systems that they wish to be included in the resulting .h5 and .fits files. This is a list of strings, where current acceptable values are only ztf. In the future, this list would also include other photometric systems such as lsst and decam. When a photometric system is present in additional_photometric_systems, photometry is calculated in these filters and appended to output files.

Doing this requires switching the headers of the fits tables from fixed lists of strings to something that must be looped over to be constructed. Changes in the hdf5_compound branch would simplify this issue.

In refine_events, the user must provide a photometric_system along with their filter_name. This allows for events to be refined for any filter in any photometric system.

A photometric_system field has also been added synthetic.generate_popsycle_config_file and to the example popsycle_config.yaml. If running run.py with this configuration file, the current commits only support adding one additional photometric system at a time, which is the one listed in the photometric_system field.

ubv_to_ztf grid files Two additional files are now included in the popsycle/data directory: ubv_to_ztf-g_grid.npz and ubv_to_ztf-r_grid.npz. These contain 2D arrays for calculating the conversion from UBV to ZTF filters. As our project continues to grow, we may want to rethink where we place and package ancillary data files. For now, this seems to be the logical place for them to live.

Transforming ubv to ztf: generate_ubv_to_ztf_grid and transform_ubv_to_ztf generate_ubv_to_ztf_grid contains the method by which the ubv_to_ztf grid files are constructed. This processes would be recreated and adapted for other photometric system. The transform_ubv_to_ztf function can now be used to go from ubv filters to ztf filters.

Updates to filt_dict The filt_dict keys have been changed to include their photometric system. There are also values for ztf with the Damineli16 reddening law present.

I've also included a few smaller additional changes:

isochrone generation Isochrones are now checked for having all possible filters from all possible photometric systems. If they do not, then a regeneration is forced.

microlens_path and popstar_path Fixed how these paths are fetched to use the inspect package instead of __file__, avoiding problems when debugging the functions into ipython.

adding more options to run.py run.py can now be run with flags that skip each of the stages of the pipeline, including --skip-galaxia, --skip-perform-pop-syn, --skip-calc-events, and --skip-refine-events. This is useful if you want to continue using the run.py architecture even after a job failed to complete, or to rerun for a series of new calc_events and refine_events with different configurations.

formatting Unfortunately I cannot seem to get my editor to stop reformatting these lines. I've went through and fixed back as many as I could find, although the change in spaces at the ends of lines does not seem fixable. My apologies for this problem.

MichaelMedford commented 4 years ago

Master has been pulled into this branch and it is, I believe, ready for merging!

MichaelMedford commented 4 years ago

Because of the merge with track_more_info, this version of popsycle won't be compatible with the current version. Will adding those columns in perform_pop_syn also make it not compatible with track_more_info?

As you pointed out, we might need to start tagging these different versions. Can we start adding those tags somewhere here?

This version of PopSyCLE would be compatible with the current version (the one that we just merged into master). If run with either additional_photometric_systems = None or additional_photometric_systems = ['ztf'], the code will work. However I agree that it is about to get quite confusing which hdf5 files have which arrays when none of them are labelled but instead use the col_idx to identify what they are supposed to be.

This is why I think we should move to a new data format for the hdf5 files that uses compound datatypes. Then there is no need to use an external dictionary that doesn't move around with the data product to identify what is in the file. All columns will be named and clear to the user and developer.

My vote would be to merge this branch into master, then I will pull that master branch into the hdf5_compound branch, and create another pull request. Then you can see how the changes in that branch resolves all of these issues.

caseylam commented 4 years ago

Okay, that sounds good to me. Should we consider this a "temporary" version then? i.e. there'll be two labeled versions of popsycle:

  1. the "old"version with hardcoded hdf5 columns (what was used for the popsycle paper).
  2. the "new" version that has the compound datatypes. this one will have all the new changes (the columns sam's added, choice of ztf filters) since nobody's yet used the intermediate version between "old" and "new", if this can be finished soon before other people want to use popsycle, that seems good to me what do you and @jluastro think?
MichaelMedford commented 4 years ago

Okay, that sounds good to me. Should we consider this a "temporary" version then? i.e. there'll be two labeled versions of popsycle:

  1. the "old"version with hardcoded hdf5 columns (what was used for the popsycle paper).
  2. the "new" version that has the compound datatypes. this one will have all the new changes (the columns sam's added, choice of ztf filters) since nobody's yet used the intermediate version between "old" and "new", if this can be finished soon before other people want to use popsycle, that seems good to me what do you and @jluastro think?

I think this is a great plan. I'l merge this version into the hdf5_compound branch and start getting it ready for its own pull request. Then we pull that one into master, and create a version number and with a tag. We can tag the commit just before track_more_info came into master as version 0.01, or 0.1 or 1. Then we can name this new version: 0.0.2, or 0.2, or 2, or whatever @caseylam and @jluastro think is best :)

MichaelMedford commented 4 years ago

Looking back, I think that commit 5155cb0 is the last commit before Sam's changes rebased onto master. This makes it the last working commit before the addition of new columns.

Also, any objections if I removed the default value for photometric_system from refine_events? I think users should have to specify this field, just like they have to for filter_name and red_law.