create dask dataframes for DC1 data with analysis flags

[jchiang87]

This should be done for the Level 2 results in

/global/cscratch1/sd/descdm/DC1/rerun/DC1-imsim-dithered/
/global/cscratch1/sd/descdm/DC1/rerun/DC1-imsim-undithered/
/global/cscratch1/sd/descdm/DC1/DC1-phoSim-3a/

at NERSC.

Here are pointers from Chris (via #desc-ci) on the dask dataframe generation script:

The simple script is
sql-query.py
in
/global/u1/c/cwalter/SSim_DC1/Notebooks/Dask
(it’s not actually in the repo)
It’s simple I was just uncommenting and commenting the various lines for the 3 cases.

[jchiang87]

PR at #45

[jchiang87]

I've created the hdf files with the dataframes for the aforementioned repositories using the make_coadd_dataframes.py script at #45. The files are in my scratch area:

[cori07] pwd -P
/global/cscratch1/sd/jchiang8/DC1_coadd_catalogs
[cori07] ls -l
total 102775776
-rw-rw---- 1 jchiang8 lsst 14817959141 Sep 22 09:37 coadd-DC1-imsim-dithered.hdf
-rw-rw---- 1 jchiang8 lsst 13428238999 Sep 22 10:15 coadd-DC1-imsim-undithered.hdf
-rw-rw---- 1 jchiang8 lsst 76996140405 Sep 22 12:59 coadd-DC1-phoSim-3a.hdf

and I've copied them to the new project area here:

/global/projecta/projectdirs/lsst/groups/SSim/DC1

@cwwalter Can we proceed with archiving the imsim-undithered and phoSim-3a repositories? (imsim-dithered I guess should wait until we can copy it somewhere for further studies).

[cwwalter]

@cwwalter Can we proceed with archiving the imsim-undithered and phoSim-3a repositories? (imsim-dithered I guess should wait until we can copy it somewhere for further studies).

Yes, but as you say let's wait on copying dithered until we know it is copied successfully given issues Tony and all are having now.

[cwwalter]

I think we can close this now correct @jchiang87?

[jchiang87]

Agreed.