Added script in bin/get_twinkles_releases.py to produce yearly release

[rbiswas4]

visits and visists in coadds.

• Data files with such lists are in ./data/releases
• Files ending with _Release.csv indicate all visits during that release
• Files ending with _Coadds.csv indicate all visits including and before that release.
• The numbering scheme uses Twinkles Release scheme as explained in Issue #454

• While we understand that year 1 can be repreocessed with coadds from year 2-10, we have not produced such coadds yet. We will look into this in future Please enter the commit message for your changes. Lines starting with '#' will be ignored, and an empty message aborts the commit. On branch Issue#454/twinkles_data_releases Changes to be committed: new file: bin/get_twinkles_releases.py new file: data/releases/year_10_Coadds.csv new file: data/releases/year_10_Release.csv new file: data/releases/year_1_Coadds.csv new file: data/releases/year_1_Release.csv new file: data/releases/year_2_Coadds.csv new file: data/releases/year_2_Release.csv new file: data/releases/year_3_Coadds.csv new file: data/releases/year_3_Release.csv new file: data/releases/year_4_Coadds.csv new file: data/releases/year_4_Release.csv new file: data/releases/year_5_Coadds.csv new file: data/releases/year_5_Release.csv new file: data/releases/year_6_Coadds.csv new file: data/releases/year_6_Release.csv new file: data/releases/year_7_Coadds.csv new file: data/releases/year_7_Release.csv new file: data/releases/year_8_Coadds.csv new file: data/releases/year_8_Release.csv new file: data/releases/year_9_Coadds.csv new file: data/releases/year_9_Release.csv

  Untracked files: bin/Failures.csv bin/InstanceCatalogs/ bin/None bin/Twinkles_1.3_first_year.csv bin/Twinkles_1.3_other_years.csv bin/Untitled.ipynb bin/check.txt bin/first_year.csv bin/get_failedvisits.py bin/minion_1016_sqlite.db bin/new.txt bin/new_w_3p4.txt bin/other_years.csv bin/phosim_input_230.txt bin/phosim_instance_catalog_220.txt bin/run_diffim.sh bin/spectra_files/ bin/splitfiles.zip bin/splitvisits.zip bin/splityears.py bin/test.txt bin/testScript.dat bin/tmp/ bin/twinkles_visits_fieldID_1427.txt data/66345752cf7ec0cc3ffa data/CatSimGals_radp3.csv data/Coadds/ data/TwinklesObsHistIDs.csv data/agn_host_id.txt data/agn_host_id.txt.old data/agn_hosts.csv data/galaxy_cache.db data/galtileid_catalog.csv data/kraken_dists.png data/lsst_e_230_f2_R22_S11_E000.fits.gz data/obsHistIDValues_run3.csv data/psf_dist.ipynb doc/LSST_DESC_Notes/0001_twinkles_design/ doc/LSST_DESC_Notes/ErrorModel.pdf doc/LSST_DESC_Notes/ErrorModel.tex doc/LSST_DESC_Notes/desc-0001-twinkles_design/ doc/calibration.md examples/CatSimGalszless1p2_radp3.csv examples/download_gals.py examples/notebooks/HostedSN.csv examples/notebooks/SimulationCheck.ipynb examples/notebooks/TwinklesPredictedTimes.png examples/notebooks/Untitled.ipynb examples/notebooks/Untitled1.ipynb examples/notebooks/Untitled2.ipynb examples/notebooks/hostLessSN.csv examples/notebooks/mytest_1 examples/notebooks/mytest_1.hdf examples/notebooks/snparams.csv examples/notebooks/stamp_0.png examples/notebooks/stamp_1.png examples/notebooks/stamp_2.png examples/notebooks/stamp_g.fits examples/notebooks/stamp_i.fits examples/notebooks/stamp_r.fits examples/notebooks/year_10_Coadds.csv examples/notebooks/year_10_Release.csv examples/notebooks/year_1_Coadds.csv examples/notebooks/year_1_Release.csv examples/notebooks/year_2_Coadds.csv examples/notebooks/year_2_Release.csv examples/notebooks/year_3_Coadds.csv examples/notebooks/year_3_Release.csv examples/notebooks/year_4_Coadds.csv examples/notebooks/year_4_Release.csv examples/notebooks/year_5_Coadds.csv examples/notebooks/year_5_Release.csv examples/notebooks/year_6_Coadds.csv examples/notebooks/year_6_Release.csv examples/notebooks/year_7_Coadds.csv examples/notebooks/year_7_Release.csv examples/notebooks/year_8_Coadds.csv examples/notebooks/year_8_Release.csv examples/notebooks/year_9_Coadds.csv examples/notebooks/year_9_Release.csv python/desc/twinkles/.obsHistIDOrdering.py.swp python/desc/twinkles/FirstSet_obsHistIDs.csv python/desc/twinkles/Untitled.ipynb python/desc/twinkles/Untitled1.ipynb python/desc/twinkles/Untitled2.ipynb python/desc/twinkles/compare_opsimMD.py python/desc/twinkles/gen_script.py python/desc/twinkles/generatePhosimInput_new.py python/desc/twinkles/ingestRepo.py python/desc/twinkles/kraken_1042_sqlite.db python/desc/twinkles/phosimInstanceCatalogs.py python/desc/twinkles/tmp/ python/desc/twinkles/twinklesSky.py setup/1 setup/2 setup/declare_eups.py setup/otherSetup.sh setup/setup_locations.sh

Fixes #454

[coveralls]

Coverage remained the same at 41.35% when pulling f81b0dc05b4b26505bb9e0c2a339598bdd632bb3 on Issue#454/twinkles_data_releases into 1502aa60b02ed4a75ec7d9f9c65d496b835fd963 on master.

[drphilmarshall]

Thanks Rahul! Looking at the numbers of visits in each release:

I think:

• The coadd image in each DR (not the diff image template/reference coadd, but the static sky measurement coadd) should contain all visits taken in that year plus all visits taken in previous years - so the DR1 coadd should contain about 2000 visits, not 0.
• The template image (not the coadd) in DRk should be made from visits drawn from years 1 through (k-1) - so the DR1 template should contain 0 images.
• We should collect the visits in DRk that are to be differenced against the DRk template in a file called year_k_DIAvisits.csv (because the "Release" should contain all image products, not just DIAimages).

Question:

• Are you including visits from both the DDF proposal and the WFD proposal? It'd be nice to make a WFD set too, so that we can make an error model for the WFD photometric SN sample as well as the DDF spectro/photometric one. I guess it's fair to include both WFD and DDF visits in a "DDF" dataset (because its not like we will ignore the WFD visits when analyzing the real DDFs!) - but the WFD one shoudl contain only WFD proposal visits. Would it be easy to make eg Year_03_WFD_DIAvisits.csv as well as Year_03_DDF_DIAvisits.csv etc?

NB. It makes the file system accounting slightly easier if you number data releases as 01, 02, 03 etc so that they ls nicely...

What do you think?

[drphilmarshall]

PS. You can remake that plot by pasting numbers into this spreadsheet. I did this:

wc -l year_?_Release.csv year_10_Release.csv | grep -v total | awk '{print $1}'

and then just pasted in the resulting column of values.

[rbiswas4]

@drphilmarshall : Now there are three files for each 'release year'. a DIAvisit file, a coadd file, and a template file. The numbering scheme includes padding for better accounting. This also means that your awk command needs '??' in place of '?' aside from the name change.

Question: Are you including visits from both the DDF proposal and the WFD proposal? It'd be nice to make a WFD set too, so that we can make an error model for the WFD photometric SN sample as well as the DDF spectro/photometric one. I guess it's fair to include both WFD and DDF visits in a "DDF" dataset (because its not like we will ignore the WFD visits when analyzing the real DDFs!) - but the WFD one shoudl contain only WFD proposal visits. Would it be easy to make eg Year_03_WFD_DIAvisits.csv as well as Year_03_DDF_DIAvisits.csv etc?

Good point. And it made me realize I had a bug (which I fixed).

I created a WFD set separately. It is possible (without a whole lot of work) to create a DDF set, but I feel we should consider WFD + DDF togther. So the two sets I made are WFD and Combined.

[coveralls]

Coverage remained the same at 41.35% when pulling 6584c845478d06be74e53d75152930fc69374fc0 on Issue#454/twinkles_data_releases into 1502aa60b02ed4a75ec7d9f9c65d496b835fd963 on master.

[drphilmarshall]

Excellent, thanks Rahul! I think this looks good - see plots below, and I'll merge this in. Thank you! :-)

@tony-johnson Some additional notes for you:

• For Twinkles 1 DR01, there are zero template visits (because template visits come from the previous years' observations), so for this DR we should only run the static sky branch of the workflow.

• The program laid out here involves 20 DM Level 2 pipeline "runs", one for each data release. I expect the later years to fail due to RAM issues with large visit numbers, so we should begin with DR01 and work our way up. It's also possible that the WFD ("wide-fast-deep", the cadence used in the main survey area rather than the deep drilling fields) runs in the early years will fail in the DIA branch, if the template visit selection cannot find any images with the right image quality... That's OK for now, I think: when we move to a better template coadd algorithm that problem should go away. So, I'm expecting us to be able to make something like WFD DR03-DR10 and Combined DR01-DR05. (Note that the number of visits in the DR10 WFD set is less than the number of visits in the DR01 Combined set!)

@tony-johnson Do you need anything else from Rahul or Simon to get started (apart from more NERSC hours and Tony-time)? cc @SimonKrughoff