Specify LSS columns

[slosar]

A recap of email thread: @egawiser @cwwalter @jchiang87 @connolly @richardxdubois @ixkael

I'm moving this to a ticket, so that it is public and remains here for posterity.

Chris Walter:

During your meeting today could you discuss the following:

We have been meeting (SSim + CI) to get the DC1 production going. We need to know what columns you would like available in the (hopefully qserve) database that will be used to store the results of the DM running.

James Chiang:

The baseline schema for the Qserv tables are here:

https://lsst-web.ncsa.illinois.edu/schema/index.php?sVer=baseline

Perhaps have a look at the Object table,

https://lsst-web.ncsa.illinois.edu/schema/index.php?sVer=baseline&t=Object

[slosar]

@cwwalter

Chris, first I need to understand a bit better what is going on here: what is the reason we don't just get the full DM output? I.e. all columns look potentially interesting and while we can make our compressed value added catalog, having the full table and ability to test-drive DB queries seem like a valuable experience.

[jchiang87]

The scripts for filling every entry in every table in the baseline schema from the Stack output do not exist as far as I know. So the reason for asking is so that we can allocate the resources from our limited manpower to implement what you think you really need.

[slosar]

Ok, I see. So, what do we get by default? If we come with some minimal list, can we go back to the stack output directly and look at it ourselves? Presumably all this lives at NERSC?

[jchiang87]

The Stack produces catalogs from the analysis of the coadds with these columns, so we could provide db table access to any of those. Nevertheless, the FITS catalogs containing those data will be available at NERSC, so you can extract what you need from them directly.

[fjaviersanchez]

From what I've seen in other issues it looks like we will have very low seeing variations (clear sky in PhoSim configuration) if any. Magnitudes from CatSim will be extinction corrected as well, right? Airmass will change in this simulation or will every target be simulated with airmass 1.0?

I think that we want, in addition to the DM columns, the extinction map consistent with the input (if any), an average airmass per exposure/target, and a map with seeing variations if there are any. And if depth variations are included at this point, we would need them as well (due to dithering or any other effect).

[cwwalter]

The airmass is in the OpSim info that drives PhoSim. Is there currently a way for us to pass that info into the final table in the work flow?

[jchiang87]

Right now the scripts that fill the db tables are run outside of our version of the Level 2 pipeline. In principle, we have control over the schemas for the tables, so we can add whatever columns we want.

[fjaviersanchez]

Yeah, an extra column would work. At the end of the day we would produce some sort of map (healpix or other) with the average airmass (or any other systematic) per pixel across the footprint, and then cross-correlate it with our signal. If you are using MAF airmass/seeing/extinction, etc. maps as an input, we could use those in our analysis without adding any extra columns. With that being said: I would prefer to have the extra columns and the maps (if you use them), the more information, the better.

[slosar]

Yes, it's great to see javier on top of this. I know it is DC1 and so really testing the testing pipeline, but at some point we need a full enumeration of all simplifications that were done wrt to everything phosim is capable of doing.

[cwwalter]

Right now the scripts that fill the db tables are run outside of our version of the Level 2 pipeline. In principle, we have control over the schemas for the tables, so we can add whatever columns we want.

Do we have the ability to connect the info in the opsim schema with the output tables in the current pipeline?

[jchiang87]

Some of the info in the opsim db file does appear in the Visit table in the baseline schema. I'm not sure that the Level 2 pipeline is the most sensible place for that information to be "connected". Filling the Visit table would be a fairly trivial task that wouldn't require any special Level 2 code.

[cwwalter]

@jchiang87 Do we need another issue to make sure that the airmass gets added to the visit table?

Also, I am a bit unclear whether we have decided in this issue if (other than the airmass issue) we are happy with the default values Jim posted from the schema.

[jchiang87]

@jchiang87 Do we need another issue to make sure that the airmass gets added to the visit table?

I don't think so. When we fill the Visit table, we'll just fill it with the relevant info from the opsim db file, and that includes airmass.

[fjaviersanchez]

The db default columns are good for us and as long as we have the visit table, and this table includes airmass, extinction, sky-brightness, seeing, and we know the dithering pattern. I guess that if we need some additional information, we will be able to get it even if it's not a straightforward process but, right now I can't think of anything else that we need.

[cwwalter]

@jchiang87 and @fjaviersanchez Are we good to close this issue now?

I think the conclusion is that we will take the default db database and add the relevant opsim info to the visit table.

[jchiang87]

yes, I think we're good.