Set maximum star magnitudes and record positions in DC1 pipeline.

[cwwalter]

Based on the discussion in issue #19 we have decided to set a maximum magnitude of 10, replacing brighter stars with this magnitude and recording for which objects and their positions this change was made. The LSS group will use this information (along with a few fully simulated bright stars) to make appropriate masks.

This issue is to record the request and make sure the feature is implemented.

[johnrpeterson]

I do think we should to revisit this in a few weeks, and do another timing analysis. In the distant past, we didn’t use any limit. Whether the optimizations and/or physics detail have changed the calculus of this, I am not sure.

But i have recently realized that the twinkles phosim runs at slac were run in a rather non-standard parallelization scheme that we didn’t intend. Basically, the runs were a factor of 2.2 less parallelized than they should have been, using a factor of 4.5 more memory than they should have been, and using a factor of 8.5 more I/O than they should have been. don’t know how exactly this will affect the total timing, but as you can see these are big factors that will make the clusters happier.

Glenn and En-Hsin are solving this elegantly by providing a script to convert from condor submission commands (which have been optimized and tested over many years) into any other submission commands (particularly slurm for nersc) which can then get embedded into the workflow tools. this then removes any ambiguity of how to run these jobs on HPC/HTC systems.

more on this shortly,

john

On Oct 12, 2016, at 9:37 PM, Chris Walter notifications@github.com<mailto:notifications@github.com> wrote:

Based on the discussion in issue #19https://github.com/DarkEnergyScienceCollaboration/SSim_DC1_Roadmap/issues/19 we have decided to set a maximum magnitude of 10, replacing brighter stars with this magnitude and recording for which objects and their positions this change was made. The LSS group will use this information (along with a few fully simulated bright stars) to make appropriate masks.

This issue is to record the request and make sure the feature is implemented.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/SSim_DC1_Roadmap/issues/22, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8vIh2TsPOL2IntuGqrApAhRm3GhFks5qzYtggaJpZM4KVZLD.

[cwwalter]

@johnrpeterson That sounds great. But I don't think we shouldn't wait for this to go ahead (especially for the initial validation tests). We are currently two weeks behind schedule. So let's start and then if a solution is found we can make things run faster later or even add back in bright stars. But, I think we should go ahead assuming we need to do this now for DC1.

[johnrpeterson]

we’ll need to wait for the actual runs for this. any initial testing or just setting the bright stars to m=10 is fine though.

john

On Oct 13, 2016, at 8:55 PM, Chris Walter notifications@github.com<mailto:notifications@github.com> wrote:

@johnrpetersonhttps://github.com/johnrpeterson That sounds great. But I don't think we shouldn't wait for this to go ahead (especially for the initial validation tests). We are currently two weeks behind schedule. So let's start and then if a solution is found we can make things run faster later or even add back in bright stars. But, I think we should go ahead assuming we need to do this now for DC1.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/SSim_DC1_Roadmap/issues/22#issuecomment-253681244, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8gb1Ft91B0EouYG8q5u_r02cE3Ldks5qztLvgaJpZM4KVZLD.

[cwwalter]

@TomGlanzman Can you remind us of the bright star cutoff criteria that was finally used and also the location of recorded information so that @fjaviersanchez and LSS members can use it for masking?

[TomGlanzman]

Scott set up a very nice script to build the instance Catalog to contained all astrophysical objects in the specified area and limiting the maximum brightness to mag=10. Those objects with magnitudes < 10 were reset to 10. In addition, a "bright_stars" catalog file was produced along side the normal catalog. It contained all objects whose magnitude was changed: the entries in this bright_stars file are the original, unaltered, entries.

[cwwalter]

OK great. @danielsf Is it small enough that we could get that checked into the SSim_DC1 Repo?

[danielsf]

@TomGlanzman Where did those catalogs get saved?

[TomGlanzman]

@danielsf First remember that it was decided we did not need to preserve the TBs of instance catalogs, preferring to regenerate them on demand, if necessary. Thus they were stored in NERSC $SCRATCH space (with a 12-week expiration). The originals created by Mustafa are located here:

/global/cscratch1/sd/mustafa/DC1-phoSim-3

These files were copied into the production account scratch space during the job execution:

/global/cscratch1/sd/desc/Pipeline-tasks/DC1-phoSim-3

Finally, Mustafa copied his entire directory tree into the HPSS system. While not convenient to recall these files, it is possible.

[cwwalter]

I think the "brightstars catalog" must be of reasonable size right? Is it still on disk somewhere? I think we need this for analysis.

[TomGlanzman]

The list of bright stars is in a file within the same directory as the instance catalog and its #includes, and subject to the same $SCRATCH expiration. @cwwalter are you requesting a new home for these files?

Here is a typical instance catalog directory listing (for obshistID 40336):

-rwxr-xr-x+ 1 desc desc 460 Feb 24 06:33 phosim_cat_40336.txt -rwxr-xr-x+ 1 desc desc 10809 Feb 24 06:34 bright_stars_40336.txt -rwxr-xr-x+ 1 desc desc 9414742 Feb 24 06:56 star_cat_40336.txt.gz -rwxr-xr-x+ 1 desc desc 764765327 Feb 24 07:01 gal_cat_40336.txt.gz -rwxr-xr-x+ 1 desc desc 3753702 Feb 24 07:01 agn_cat_40336.txt.gz*

[danielsf]

Thanks, Tom.

If we really are serious about saving a list of bright stars in the repository, it probably makes more sense to write a separate code that just lists all of the mag<10 sources in the DC1 area (rather than keeping each bright star list from each pointing).

[cwwalter]

If we really are serious about saving a list of bright stars in the repository, it probably makes more sense to write a separate code that just lists all of the mag<10 sources in the DC1 area (rather than keeping each bright star list from each pointing).

Yes, this is what I meant.

@fjaviersanchez should chime in on what is really needed.

[danielsf]

What information do we need on each source?

RA Dec magnitude (in all 6 bands?)

anything else?

[jchiang87]

FWIW, I am already storing the true positions and magnitudes for the stars in the imsim input object catalogs:

MySQL [DESC_DC1_Level_2]> describe StarTruth;
+---------+------------+------+-----+---------+-------+
| Field   | Type       | Null | Key | Default | Extra |
+---------+------------+------+-----+---------+-------+
| id      | bigint(20) | NO   | PRI | NULL    |       |
| raICRS  | double     | YES  |     | NULL    |       |
| decICRS | double     | YES  |     | NULL    |       |
| u_mag   | float      | YES  |     | NULL    |       |
| g_mag   | float      | YES  |     | NULL    |       |
| r_mag   | float      | YES  |     | NULL    |       |
| i_mag   | float      | YES  |     | NULL    |       |
| z_mag   | float      | YES  |     | NULL    |       |
| y_mag   | float      | YES  |     | NULL    |       |
+---------+------------+------+-----+---------+-------+
9 rows in set (0.00 sec)

MySQL [DESC_DC1_Level_2]> select count(*) from StarTruth where r_mag<10;
+----------+
| count(*) |
+----------+
|      524 |
+----------+
1 row in set (1.57 sec)

MySQL [DESC_DC1_Level_2]> 

So to the extent that the input catalogs are the same for phosim and imsim, I think this info is already persisted and available in a serviceable format.

[danielsf]

At the risk of being pedantic: the "bright star/not a bright star" cut in the PhoSim InstanceCatalogs is made on the magNorm, rather than any specific LSST band. There's probably a pretty good mapping between the two, but it may not be perfect. I will generate a catalog of sources with r<10 and magNorm<10 and see how they line up.

[jchiang87]

It sounds like it would be useful to add magNorm as a column to this table.

[cwwalter]

And maybe a flag that the reassignment was made?

[jchiang87]

I forgot that the phosim catalogs reassigned the magnitudes for the bright objects. In that case, I would recommend a separate table (with the same schema) just for the phosim stars with the reassigned magnitudes. Trying to stuff that information into a single table for both the imsim and phosim stars would be unwieldy.

[cwwalter]

I thought we did the reassignment for both?

[jchiang87]

Just for phosim, since imsim didn't have the execution time issues for the bright stars.

[cwwalter]

I thought I remembered doing it for consistency (or at least talking about doing it). But, I could certainly be wrong.

[danielsf]

The bright star catalog is only about 1300 lines (and there are sources with magNorm<10, rmag>10).

I will make a pull request to add it to the SSim_DC1 repo.

[cwwalter]

This was added to the repository by Scott. Closing this issue.