DC1 phoSim configuration

[TomGlanzman]

For the record, below is the current command/override file being used in the DC1 tests (and as used in Twinkles). It was also presented at last Monday's CI meeting.

cleardefects clearclouds airglowvariation 0 fringing 0 contaminationmode 0

I have heard Chris say that using the same set of settings as Twinkles is a good thing. But John (and others?) have suggested turning on various elements, at least partially due to our using phoSim v3.6.0. What shall it be?

[TomGlanzman]

Another aspect of phoSim config, the instanceCatalogs, has seen recent progress. Scott provided me with the python to dynamically generate instanceCatalogs, a copy of which is at NERSC here:

/global/projecta/projectdirs/lsst/production/DC1/DC1-phoSim-2/config/generateDc1InstCat.py

In addition, he also pointed me to a copy of the opSim database and I have placed a copy at NERSC here:

/global/projecta/projectdirs/lsst/production/DC1/catalogs/minion_1016_sqlite_new_dithers.db.gz

(This came via the 20 Jan 2017 entry by Lynne: https://github.com/lsst/sims_operations/pull/55)

All that is left is a list of obsHistIDs with which to generate the instanceCatalogs. Is that list available somewhere @cwwalter or @danielsf or @jchiang87 ?

[jchiang87]

My understanding was that the LSS group wants us to simulate all visits associated with the 15 fieldIDs listed at https://github.com/LSSTDESC/SSim_DC1/issues/23#issuecomment-273535361. @humnaawan @cwwalter Can you confirm that?

If so, I can provide the list of visits/obsHistIDs.

[egawiser]

That sounds right, although we only want to simulate certain chips from each of those visits (which @jchiang87 has been helping us with). It's still not clear to me what "instanceCatalog" really means; we want to be sure that each instanceCatalog includes all objects that would fall within the DC1 simulation region, which is a subset of all objects that fall within the dithered LSST FOV corresponding to a given obsHistID.

[jchiang87]

Eric's comment reminds me that the list of visits (as well as the chips to simulate per visit) are already available from Humna's calculations. She gives links to those results at a later comment in the issue I referenced: https://github.com/LSSTDESC/SSim_DC1/issues/23#issuecomment-275025056 . So that should be what you need Tom, to do the dithered simulations.

[cwwalter]

Yes.. on Jim and Eric's comments above on what we want for the dithered run.

For the config file. We agreed @johnrpeterson would check if clouds were OK for photometry variations across a chip and he confirmed there was no problem with 3.6. So, that at least should be removed.

I believe contamination was added because Twinkles saw the dust etc and DM wasn't dealing properly with it yet and it required two exposures to remove them. I assume the same is true for other variations (like sky glow) So, I am fine with keeping those other options.

The important point related to above is that I believe we are doing one 30 second exposure like Twinkles. Tom, can you confirm that is how the latest Twinkles run was configured?

Also, we decided at that Oxford meeting that we would turn off the other sensor effects except for saturation. Specifically, we don't have a good correction for brighter fatter in DM yet so we want that turned off in this first simplest data challenge.

At least in the past we would do this with:

chargesharing 0

A more complete pixel charge sharing reduction would be:

chargediffusion 0 pixelerror 0 fieldanisotropy 0 chargesharing 0

The same issue is true for tree rings, but I'm pretty sure there is no option for this, it is just handled through the doping variation specified in the silicon text file. Is that correct John?

[humnaawan]

@jchiang87 as I mentioned in the telecon yesterday, we realized that covering visits to all 15 fields isn't necessary since we really only care about the area covered by 4 undithered contiguous fields. So for the chip finding part, we decided to consider a region more compact than the disc I had initially referred to, i.e. we look at chips that fall within the black outline here:

This reduces the total chips that will need to be simulated from 203,804 to ~ 131,000 for a dithered survey (131,052 when we have both random translation and rotational dithers; 131,004 with random translation dithers only).

The new output is still in the output folder in the repo, with the filename containing the tags that identify the exact details. Each pickle contains four arrays (obsHistID, expiate, fieldIDs, chipNames). Currently there are three files in the output directory.

Please let me know if anything's unclear.

[humnaawan]

Also, for debugging purposes, I was wondering if you have a chip-per-visit list for the undithered survey. If yes, can we compare that with what I have for the region as that should be a good check on the code?

[jchiang87]

For the undithered survey, I'm simply doing full focal plane simulations (189 science sensors) for each of the four fields. In the absence of dithering, I would expect your code to return all sensors for every visit.

Tom will be running phosim using the new chip-per-visit data that you provided today, so I'm sure he will ask for clarification if needed.

[johnrpeterson]

Chris & Tom-

I’m going to combine two threads here.

Chris, cleardefects covers all of the below (except chargediffusion which you definitely do not want off— otherwise there is no sensor)

The twinkles runs and DC1 tests used these:

cleardefects clearclouds airglowvariation 0 fringing 0 contaminationmode 0

Let me explain each one:

1) “clearclouds" was used to get lower opacities than what was in v3.4. There extinctions were a factor of 7 too high, due to a misinterpretation of some site measurements. This is a really, really old resolved issue though. This was already fully fixed in v3.5. And now we are on to v3.6. We have also already verified that any limitation in DM zeropointing algorithms would not have an issue with this. So I don’t see an argument for retaining this.

2) "airglowvariation 0” I don’t know why this was off. This is spatial variation of the background of one component. I suspect this was a concern about DM zeropointing algorithms, but that is not an issue.

3) “contaminationmode 0” This turns off contamination (dust/condensation) on all surfaces. In v3.4 there was no contamination on. In v3.5 there was, so this was turned off for twinkles because although the overall transmission was correct, the morphology of the dust transmission was particularly crude (leading to blocky looking absorption). This could have still been used if you were dividing by a flat (as it would cancel), but this was simply turned off. This is fully fixed in v3.6, so should be back on.

4) “cleardefects” and “fringing 0” would turn off complications of the sensor. These were not off due to any PhoSim limitation or error, but always because of a concern about DM handling these details at this point in their software development. These are all relatively subtle, and honestly the past few decades of astronomy had pipelines that didn’t correct for these things properly and everyone was ok, so you can make the argument that these won’t cause any serious problem even if they are not fully removed. On the other hand, you could always keep things simpler as many have argued. (note: i am really, really scientifically curious to see their subtle effect in a data challenge).

5) Note this is all a very small list. Everything else has been on. And note that the flats that En-Hsin has recently generated at NERSC would be compatible formally if and only if you removed all 5 (although clouds & airglow would have no effect), so we’d have to make some new ones otherwise.

Anyways, those are the technical reasons behind this. Whatever is most useful for everyone should be what is done, obviously.

john

On Jan 29, 2017, at 4:31 PM, Chris Walter notifications@github.com<mailto:notifications@github.com> wrote:

Yes.. on Jim and Eric's comments above on what we want for the dithered run.

For the config file. We agreed @johnrpetersonhttps://github.com/johnrpeterson would check if clouds were OK for photometry variations across a chip and he confirmed there was no problem with 3.6. So, that at least should be removed.

I believe contamination was added because Twinkles saw the dust etc and DM wasn't dealing properly with it yet and it required two exposures to remove them. I assume the same is true for other variations (like sky glow) So, I am fine with keeping those other options.

The important point related to above is that I believe we are doing one 30 second exposure like Twinkles. Tom, can you confirm that is how the latest Twinkles run was configured?

Also, we decided at that Oxford meeting that we would turn off the other sensor effects except for saturation. Specifically, we don't have a good correction for brighter fatter in DM yet so we want that turned off in this first simplest data challenge.

At least in the past we would do this with:

chargesharing 0

A more complete pixel charge sharing reduction would be:

chargediffusion 0 pixelerror 0 fieldanisotropy 0 chargesharing 0

The same issue is true for tree rings, but I'm pretty sure there is no option for this, it is just handled through the doping variation specified in the silicon text file. Is that correct John?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-275947773, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8itYv-DKE-vvsxnyqWwFVLysw2QPks5rXQUggaJpZM4LwP6n.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[TomGlanzman]

@cwwalter - yes, twinkles ran with single 30s exposures per visit

@johnrpeterson and @cwwalter thanks for your comments on the command/override file. Are we converging on the following:

cleardefects fringing 0 chargesharing 0

or do we also want to add:

pixelerror 0 fieldanisotropy 0 chargesharing 0

[johnrpeterson]

cleardefects includes chargesharing, pixelerror, field anistropy. so no.

john

On Feb 1, 2017, at 1:22 PM, Tom Glanzman notifications@github.com<mailto:notifications@github.com> wrote:

@cwwalterhttps://github.com/cwwalter - yes, twinkles ran with single 30s exposures per visit

@johnrpetersonhttps://github.com/johnrpeterson and @cwwalterhttps://github.com/cwwalter thanks for your comments on the command/override file. Are we converging on the following:

cleardefects fringing 0 chargesharing 0

or do we also want to add:

pixelerror 0 fieldanisotropy 0 chargesharing 0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-276737654, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8k-qGFTC38vkrhCy1_kVuDzAikgCks5rYM1ygaJpZM4LwP6n.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[cwwalter]

Hi John,

Will cleardefects clear tree rings?

[johnrpeterson]

yes.

by the way, i still think you can make the argument either way of whether these defects should be kept on. take tree rings for example. there is no complete correction in DM now. if you do the traditional thing in astronomy analysis and divide by the flats as well as do some local astrometric empirical fitting, you’d not really correct them correctly, but you’d kind of get the signature removed at the ~halfway level approx. On the other hand, if you just remove them in phosim you’d be underestimating your errors by a similar amount because the simulations will be too easy. so on would overestimate errors and off would underestimate by a ~similar amount. just a thought.

john

On Feb 1, 2017, at 1:36 PM, Chris Walter notifications@github.com<mailto:notifications@github.com> wrote:

Hi John,

Will cleardefects clear tree rings?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-276741299, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8hdWP31UjDHA0ZO0HH3uWRzdfWZLks5rYNCmgaJpZM4LwP6n.

[TomGlanzman]

@humnaawan thank you for the DC1 visit data files. I would like to confirm that corner raft sensors do appear in your lists... and that I may safely ignore them?

For @cwwalter and and others, which of these three configurations should be used for DC1, with or without field dither, and with or without rotational dither?

[cwwalter]

I thought Humna removed the wavefront sensors, but you can ignore them. We will not simulate those.

We want the full translational + rotational dither.

[TomGlanzman]

Thanks @cwwalter , so that will be: 2017-01-29_chipPerVisitData_newAfterburnerOutput_fID1447_RandomDitherFieldPerVisit_randomRotDithered_nonDiscRegion_131052TotChipsToSimulate.pickle

which contains 2395 visits.

[cwwalter]

That you will have to check with Humna on :)

[johnrpeterson]

did you guys pick up that you should do “camconfig 1” for this?

[TomGlanzman]

Yes, @johnrpeterson , thanks we will use camconfig 1

[TomGlanzman]

Here is the full current working version of the command/override file. Comments welcome...

# Enable centroid file centroidfile 1

# Disable sensor effects cleardefects fringing 0

# Set the nominal dark sky brightness zenith_v 21.8

# Science sensors only camconfig 1

[johnrpeterson]

camconfig 1 is in the catalog not the command file.

[TomGlanzman]

Oops sorry, you're right. (And it is there in the instance catalog.)

[cwwalter]

Is the zenith_v command still necessary? I understand that was due to a bug found in the earlier twinkles run?

[sethdigel]

I took a look at 2017-01-29_chipPerVisitData_newAfterburnerOutput_fID1447_RandomDitherFieldPerVisit_randomRotDithered_nonDiscRegion_131052TotChipsToSimulate.pickle tonight, to look up the observation metadata. The pickle file contains 14 field IDs, with this distribution of counts of obsHistIDs:

Field ID	# observations
1212	136
1220	161
1234	141
1305	178
1323	202
1333	201
1365	160
1413	165
1431	202
1447	202
1464	186
1542	150
1564	176
1568	135

I have not been following this thread very closely, but I had thought that the file would have four field IDs.

Here is the distribution of moon altitudes for the 2395 visits.

I haven't tried to estimate the CPU time required in this multi-threaded era, but 986 of the visits have positive moon altitude, which for the Twinkles phoSim visits has corresponded to large CPU times.

[jchiang87]

It would only have four fieldIDs if there were no translational dithering. The target region is defined by four fields in the undithered case. In the dithered case, 14 (once 15?) fields are translated to overlap with the target region.

[johnrpeterson]

it wasn’t a bug but a limitation. but chris you should investigate this with the cadence of DC1.

the issue is that if you do 2 different phosim runs closely separated in time it may have very different parameters (like zenith_v). in reality it is highly correlated on ~minute time scale. so twinkles just fixed the background for all runs. this is being dealt with comprehensively for v3.7, but this limitation is still there in v3.6.

however, this is probably irrelevant for DC1 as there exposure times are probably all over the 10 year period and its focus is not on variability. can you check this?

john

On Feb 1, 2017, at 4:55 PM, Chris Walter notifications@github.com<mailto:notifications@github.com> wrote:

Is the zenith_v command still necessary? I understand that was due to a bug found in the earlier twinkles run?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-276795296, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8sKtpLfM8YSRIoszLclDnsiI9ObDks5rYP84gaJpZM4LwP6n.

[cwwalter]

the issue is that if you do 2 different phosim runs closely separated in time it may have very different parameters (like zenith_v). in reality it is highly correlated on ~minute time scale. so twinkles just fixed the background for all runs. this is being dealt with comprehensively for v3.7, but this limitation is still there in v3.6.

however, this is probably irrelevant for DC1 as there exposure times are probably all over the 10 year period and its focus is not on variability. can you check this?

It is true that we don't care about variability here and the stack is going to take out overall levels on each chip. So, it sounds like it is fine to keep it.

But, just to make sure I understand: OpSim should of course include these correlations as it moves from visit to visit. So, to the extent that OpSim drives the catalog you should get correlations. On the other hand, if you restrict your self to visits in some small area out of a full OpSim survey the 'last' visit might have actually been a long time ago. So, that correlation can be lost. However, that effect seems to be the opposite direction of what you suggest above so can you be a bit more explicit?

Thanks.

[TomGlanzman]

Below is a sample dynamically generated instanceCatalog (sans the included objects). Please review.

rightascension 89.1788674 declination -30.1024577 mjd 59634.0910186 altitude 67.2676473 azimuth 263.6360252 filter 2 rotskypos 5.2752167 camconfig 1 dist2moon 122.9044891 moonalt -19.2875356 moondec -21.8515217 moonphase 47.7539710 moonra 246.1223288 nsnap 1 obshistid 40336 rottelpos 102.3316946 seed 40336 seeing 0.5462840 sunalt -33.7896703 vistime 30.0000000 includeobj star_cat_40336.txt includeobj gal_cat_40336.txt includeobj agn_cat_40336.txt

[SimonKrughoff]

@TomGlanzman I would vote for adding contaminationmode 0 for two reasons:

1. I haven't seen a v3.6 flatfield yet or any validation from the phosim team that this is no longer an issue.
2. (related) We don't have flats yet, so will be doing processing using the eimages. It's a bit circular, but processing the eimages means we can't apply flats.

I'm always a little worried about turning things on because we often get surprised, so it would be good to produce a few images to look at before producing a whole ton of them.

[sethdigel]

Back on dithering, here is a plot showing the positions of the 14 fields in 2017-01-29_chipPerVisitData_newAfterburnerOutput_fID1447_RandomDitherFieldPerVisit_randomRotDithered_nonDiscRegion_131052TotChipsToSimulate.pickle and the positions of the dithered observations as defined in minion_1016_sqlite_new_dithers.db. The instance catalogs that Tom is generating draw the dithered positions from the quantities randomDitherFieldPerVisitRA and randomDitherFieldPerVisitDec in that database.

I guess that the region defined in Humna's diagram above is the central four fields. I'd guess that each of the dithered visits would have some overlap with the central region, although possibly not much at all, for the visits on the outskirts dithered away from the central region.

I've added minion_DC1.csv to the data directory in the Twinkles branch u/digel/dc1_dithers that has the coordinates and some of the observation metadata for the DC1 visits. In that file, the columns 'ra' and 'dec' are just the coordinates of the field centers.

[egawiser]

Thanks, Seth! That's a very nice illustration of the visits that are chosen because at least one CCD overlaps with the region covered by the 4 central (undithered) fields. The "tilt" of those 4 is correct, although it looks opposite in our plots which show RA increasing towards the left (standard astronomical convention of looking up at the sky). The instance catalogs should presumably also use the rotationally-dithered versions of rotSkyPos and rotTelPos, as discussed yesterday in #Issue#55.

[johnrpeterson]

looks good.

On Feb 2, 2017, at 1:22 PM, Tom Glanzman notifications@github.com<mailto:notifications@github.com> wrote:

Below is a sample dynamically generated instanceCatalog (sans the included objects). Please review.

rightascension 89.1788674 declination -30.1024577 mjd 59634.0910186 altitude 67.2676473 azimuth 263.6360252 filter 2 rotskypos 5.2752167 camconfig 1 dist2moon 122.9044891 moonalt -19.2875356 moondec -21.8515217 moonphase 47.7539710 moonra 246.1223288 nsnap 1 obshistid 40336 rottelpos 102.3316946 seed 40336 seeing 0.5462840 sunalt -33.7896703 vistime 30.0000000 includeobj star_cat_40336.txt includeobj gal_cat_40336.txt includeobj agn_cat_40336.txt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-277039167, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8ul8_HY7cJiQwjFSGXcoMbLWz5acks5rYh71gaJpZM4LwP6n.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[johnrpeterson]

On Feb 2, 2017, at 11:51 AM, Chris Walter notifications@github.com<mailto:notifications@github.com> wrote:

the issue is that if you do 2 different phosim runs closely separated in time it may have very different parameters (like zenith_v). in reality it is highly correlated on ~minute time scale. so twinkles just fixed the background for all runs. this is being dealt with comprehensively for v3.7, but this limitation is still there in v3.6.

however, this is probably irrelevant for DC1 as there exposure times are probably all over the 10 year period and its focus is not on variability. can you check this?

It is true that we don't care about variability here and the stack is going to take out overall levels on each chip. So, it sounds like it is fine to keep it.

But, just to make sure I understand: OpSim should of course include these correlations as it moves from visit to visit. So, to the extent that OpSim drives the catalog you should get correlations. On the other hand, if you restrict your self to visits in some small area out of a full OpSim survey the 'last' visit might have actually been a long time ago. So, that correlation can be lost. However, that effect seems to be the opposite direction of what you suggest above so can you be a bit more explicit?

Thanks.

yes, what you say is correct. PhoSim naturally picks up correlations of observing parameters. but this is fairly irrelevant as you say if you just pick a half dozen fields over 10 years.

so all that is fine. for the most important things (like say position of the moon), all versions of phosim naturally pick up these behaviours and it simulates it properly.

so i am only discussing the more minor second order things like the dark sky background variation (which is very small compared to the variation of whether the moon is there or not). those things the opsim scheduler wouldn’t optimize on, and therefore phosim simulates them from the distribution. however, as i was saying the short time correlations (~minutes to ~hours) are not in phosim until v3.7. so you can either fix it to a single value which is what we did for twinkles or accept that two exposures closely spaced in time will not have the time correlations.

i am fairly certain as you are that this is irrelevant for LSS studies for DC1.

john

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-277013700, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8jr8moNaF1TZPgOb-2c24oWC-XZSks5rYgmTgaJpZM4LwP6n.

[sethdigel]

Thanks, Seth! That's a very nice illustration of the visits that are chosen because at least one CCD overlaps with the region covered by the 4 central (undithered) fields. The "tilt" of those 4 is correct, although it looks opposite in our plots which show RA increasing towards the left (standard astronomical convention of looking up at the sky). The instance catalogs should presumably also use the rotationally-dithered versions of rotSkyPos and rotTelPos, as discussed yesterday in #Issue#55.

I'm mortified for having reversed the RA axis. I've fixed it. Yes, Scott's script that Tom is using to make the instance catalogs also modifies rotSkyPos and rotTelPos.

[TomGlanzman]

Is there any objection to implementing Simon's suggestion, "contaminationmode 0"? If not, I will make the change for the next round of test runs.

(Nice plot Seth!)

[johnrpeterson]

tom, its all one word “contaminationmode 0” (as all phosim commands are).

john

On Feb 3, 2017, at 12:08 PM, Tom Glanzman notifications@github.com<mailto:notifications@github.com> wrote:

Is there any objection to implementing Simon's suggestion, "contamination mode 0"? If not, I will make the change for the next round of test runs.

(Nice plot Seth!)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/SSim_DC1/issues/27#issuecomment-277304474, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8nXeXtStSL1una8CXUjB-kqiZ7J9ks5rY18fgaJpZM4LwP6n.

[TomGlanzman]

Right, I noticed and edited the github issue entry immediately, but that did not trigger an updated email. - Tom

On 2/3/17 09:23, johnrpeterson wrote:

tom, its all one word “contaminationmode 0” (as all phosim commands are).

[TomGlanzman]

No objections to reinstating the 'contaminationmode 0' directive, so it is back in.

Unless someone has other configuration concerns, let's call this a wrap. Over the next couple of days, my plan is to use this configuration to generate a number of full focal plane visits. Everyone will be welcome to inspect the results prior to ramping up production.