Sharing DC2 data with WWT

[heather999]

We have agreed to provide DC2 data to the World Wide Telescope, with the understanding that the data will be used for this technical demonstration only and not for science or made public. Specifically, the goal is "Demonstration of visualization capability".
From Andy:

He was interested in trying to visualize some of the LSST simulated data from DC2 (possibly converting the fits files into color images and loading them into WWT) as a demonstration of WWT’s capabilities. Would you be able to help Peter get hold of some of the images. I would assume 1-2 fields on the sky with a few visits per pointing (so there is color and time information).

I can help set this up so they can access the files, but need some ideas of specifically what data we want to share. Hoping to get some help from @jchiang87 and @erykoff

[jchiang87]

Given Andy's suggestion, I think the development dataset we defined for gen3 work would fit the description: https://github.com/LSSTDESC/gen3_workflow/wiki/Development-Datasets I would think that WWT may also want deep coadds to make color images. In that case, we could point them at the tract 3828 gri color image I posted in Slack using DR6 coadds, as an example.

[heather999]

The WWT folks wrote back:

Deep coadds and/or color images would be very useful to have, thanks.

I've opened a ticket with NERSC to set up a shared globus endpoint so we can provide easy access to the data. I can start linking in the gen3 dev dataset in this area.

[heather999]

The globus endpoint has been set up and sent to the WWT folks. The area contains the 30 visits in that development dataset and the WWT folks confirmed they can see it via Globus.
As for deep coadds/color images - what specifically do we want to provide? I can drop that tract 3828 gri color image in the same area - is there anything else to include?

[jchiang87]

Can we just set up an email thread with whomever you are in contact with so we can have a direct conversation?

[jchiang87]

Based on Peter's preference for time-variable data, I'd recommend providing the calexps and warped images (excluding the psfMatchedWarp* files) for the gri bands of the DR2 tract 5063 data. I'd also include the assembled coadds for those bands.

[heather999]

Just to follow up @jchiang87 when you say gri bands of DR2 tract 5063, I am assuming that would be the down-selected Run3.1i data and waiting for the DR2 processing to obtain those assembled coadds and any missing warps. Correct? Given the ongoing issues with NERSC, I'd like to follow up with Peter to make sure he knows he has not been forgotten.

[jchiang87]

Yes, exactly, the DR2 dataset we've been preparing.

[heather999]

With CSCRATCH back, I went through the track_mapping.sqlite3 for DR2 and found 73 visits from bands gri tract 5063 (does that sound approximately right?). I've copied the calexps for now (just the calexp files - I can grab other data files under dr2-calexp if needed) in: /global/cfs/cdirs/lsst/gsharing/DC2_ImSim/DR2/repo/rerun/dr2-calexp/calexp For organizational purposes I'm keeping the DM repo structure - but if it's better to reorganize this somehow for the WWT folks, just let me know. I'll wait for DR2 to be finished and validated before grabbing the warps and assembled coadds.
Is it useful to WWT to get just the calexps for now?

[jchiang87]

Is it useful to WWT to get just the calexps for now?

Based on Peter's most recent emails, it sounds like the warped images, coadds, and eventually, the difference images are what he'd find most useful. It wouldn't hurt to let him know about the calexps, but I wouldn't think the other data files in dr2-calexp are useful to him.

[heather999]

Just to follow up on Peter's most recent email - it sounds like he received the data he needs though I'm not clear how. Does he have the DR2 data discussed above?

[jchiang87]

I have no idea how he got those data. I assumed it was through the Globus endpoint you set up. Is there anything available at that location now? Where can we see that area at NERSC? Based on what he showed, it is not really what we discussed with regard to warps and coadds.

[heather999]

Ok - yes, I see with his link.. He seems to be using the initial data I set up of just those 30 visits from Run2.2i sim Y1-wfd which are available in /global/cfs/cdirs/lsst/gsharing/DC2_ImSim/Run2.2i/sim/y1-wfd I have those calexps for tract 5063 bands gri from DR2 in place for Globus sharing - and I could just grab the warps since I don't believe any part of 5063 is in the list of corrupt warp files -similarly, the coadds might be ok too. Would we want to look at that more deeply before sharing or just indicate that this is preliminary?

[jchiang87]

I think can just go ahead and share the warps and coadd files with him and let him know that those data are still preliminary.

[heather999]

Finally, I have the warps and coadd files for tract 5063, band gri, set up in /global/cfs/cdirs/lsst/gsharing/DC2_ImSim/DR2/repo/rerun/dr2-coadd I grabbed the contents of deepCoadd-results and deepCoadd, excluding the psfMatchedWarp* files. If that looks ok, we can send an update to Peter.

[jchiang87]

Looks good to me. I'm not sure that the deepCoadd-results folder is necessary since the coadded images are in deepCoadd, but Peter may find the calibrated versions useful.

[katrinheitmann]

Have we heard anything more about this? Or should we treat this as done?

[heather999]

We last communicated with Peter Williams on November 9th. Just for completeness I'll add some additional details communicated in those emails and a summary. We could try following up with him.

There is a Globus endpoint (dedicated to this purpose) set up at NERSC and only accessible by Peter. We made available:

• 30 simulated visits from DC2 Run2.2i Year 1 WFD, 5 visits from each band grizyu. https://github.com/LSSTDESC/gen3_workflow/wiki/Development-Datasets
• coadds and warps for DC2 year 1 tract 5063 bands gri

Peter provided a link to his demo which was set up using just the simulated visits - I'm not sure if he has downloaded and used the coadds and warps. https://newton.cx/~peter/wwt-lsst-fall2020/ source code: https://github.com/pkgw/wwt-lsst-fall2020/

This demo has processed the underlying FITS data into RGB images, so you can't (e.g.) change the colormapping like in DS9. AAS WorldWide Telescope can render single FITS files, though, and we're working on mashing up these functionalities so that you can explore a huge image like an LSST tract with on-the-fly colormapping and all the rest.

If, having seen my current demo, there's any kind of "explorer" UI that you'd enjoy playing around with, please let me know — you can achieve a lot of fancy-looking effects pretty easily with these modern web toolkits!

[katrinheitmann]

Visualization was shared, no more interactions after that.