choose night of the survey for initial detection catalog

[rhiannonlynne]

@stevechesley (but to be done after issue #1)

[rhiannonlynne]

full SSM available MJD 50093 to 51893 (+30)

In the opsim databases 'night' corresponds to the night since the start of the survey (starting with 0).

night = 747 corresponds to exposure times ranging from 50100.031278 to 50100.375172 and the visits are only for WFD (Universal cadence, 'main survey') and NES (North Ecliptic Spur), plus they are only taken in r and z band (650 in r, 106 in z -- the z band observations are only in -12 to -18 degree twilight at the start and end of the night).

However, it does look like the non-pairs rate in this night may be a bit high .. lots of triples, quads, and 13% singletons. sqlite> select nobs, count(), count()_nobs from (select night, FieldfieldID, count() as nobs from ObsHistory where night=747 group by Field_fieldID) group by nobs; 1|99|99 2|102|204 3|30|90 4|44|176 5|11|55 6|13|78 7|4|28 8|2|16 10|1|10

Still, this may be a reasonable night to use to generate a 'night' of detections for initial comparison purposes. The full month we end up choosing later may be different anyway.

[stevechesley]

OK, so we will use enigma_1189, night 747. Just for validating generic generation of detections. And to compare the astrometric accuracies.

And we'll use the Grav (Bottke) NEOs with step function detection cutoff at the field-specified faint limit. No rate cutoff. We will use the "red" curve for trailing loss. Can you use a circular detection region with specified radius in degrees? And what should that radius be?

Use the H-G system with G=0.15 for all bodies. Do we need to specify the (generic) asteroid color indices that we are using to compute the apparent magnitude?

Outputs: body ID time field ID RA/DEC (&rates) apparent mag.

Anything else that might be handy for resolving discrepancies?

[rhiannonlynne]

Yes - Grav/Bottke model (there may be some small differences in our input model, due to me propagating our model to 1994 to match the survey, but I think this should be minor).

No rate cutoff. Add red curve for trailing loss - I call this the 'detection' loss, as it's related to the detection software. The SNR losses are described more appropriately by the blue curve, which I tend to call trailing loss (but am open to a less ambiguous name, maybe SNR losses).

I can use a circular detection region; I suggest a radius of 1.75 degrees. This is smaller than the outer edge of our fov but it's a reasonable radius to use because it approximately balances the chips that are outside this circle with the corner gaps that are inside the circle.

Yes - We used OpenOrb to generate the V magnitudes, which uses H-G and we set G=0.15, so this matches. We do need to specify the color indices to calculate the magnitudes in the filters - we could use a distribution of C and S objects (I'll check on what our distribution actually is).

It might be handy to report the V band magnitudes as well as the apparent magnitude in the filter (with 'detection' losses), as well as the actual RA/Dec values and rates before adding astrometric error. I'm not entirely certain how to determine astrometric error in the case of trailing; without trailing it's fairly straightforward. @mjuric @stevechesley do you have any suggestions of how to calculate astrometric error in the position for a trailed object?

Do we want to estimate a trail length and position angle? Presumably these would be important for MOPS but would eventually also have errors associated. Perhaps the errors here are not as important?

[pveres]

I run a test on night 747 and Grav's NEO model and investigated how long does it take to generate detections. It took 7.4 min on our cluster. Run on Hawaiian cluster for 10-years of data and NEOs took ~15 days to generate detections, and only 1.5 days to make tracklets.

Currently, I am running the same night on full NEO + MB population (14 million objects).

PS: how do we transform V mag to the LSST filter? In PS1 MOPS there is only a simple conversion V-"X" where "X" is the filter. I am using PS1 filters (similar to Sloan and LSST) for this study. For an average S+C asteroid, the transformation (derived by Fitzsimmons) is:

u = -0.3 (this one is only a guess) g = -0.28 r = 0.23 i = 0.39 z = 0.37 y = 0.36

I also have a table for asteroid classes from Alan Fitzsimmons (see table 2 page 12 in H&G paper): http://arxiv.org/pdf/1506.00762v2.pdf

This is important to know because of the magnitude limits. We should use the same transformations in our studies.

[mjuric]

@rhiannonlynne Re astrometric error of trailed objects -- good question, we should see if we can simulate it (and derive it from first principles).

My _guess_ is that the error in the direction perpendicular to the trail will be approximately the same as the error for a stationary object with the same SNR integrated over the trail.

In the direction along the trail, it's really only the endpoints that matter (you're trying to assess where the midpoint is). So I'd guess that the error will be approximately equal to that of a stationary object with an SNR equal to detected SNR (the SNR we get by just convolving the trail just with the PSF -- i.e., the one that the detection pipeline reports).

[stevechesley]

GitHub question: Does putting the @username in the message do something special, like generate an email? So if I want everybody to see this I need to put @mjuric @rhiannonlynne @pveres somewhere?

Trailed astrometric errors: If you are integrating the signal along the trail through trail fitting then I expect you would get a cross-trail error given by the integrated SNR. The along trail error would be obtained by convolving a semicircular PSF at the end of the trail, right? Is that the same as using a circular PSF somewhere near the middle of the trail?

BUT: For the purposes of validation on this single night we want to remove statistical variations, and so we want to see the exact photometric and astrometric predictions. Maybe noise could be included as a separate column?

[mjuric]

@stevechesley: yes, @-ing someone ensures they get an e-mail notification even if they're not following the issue. It's also useful to disambiguate to whom one is answering when there are multiple answers one wants to give in a single reply.

Re astrometric errors: 100% agreed -- you've phrased it much better than I did. Semi-circular PSFs would be the most accurate way to do it, though my guess is that using a circular PSF somewhere near the middle may not be a bad approximation (it will overestimate the accuracy somewhat, that's true). The nice thing about the latter is that that's the number we get right now out of the object detection pipeline.

[stevechesley]

@rhiannonlynne What color indices are you using? This is with catsim, and so you have two different color vectors (V - filter_i) for the two different taxonomic types? Can you treat all the asteroids the same? Or tell us which objects are which type?

[rhiannonlynne]

@pveres I'm generating a list of NEO detections for the single night now, and could run for the 10 years of the survey next .. I need a little more time to get set up to do the full solar system (sorry - got to dig up some old scripts and add the trailing loss). [aside: my NEO-only detections are generated using the MafSSO code; the full solar system detections will be generated using CatSim code, so this will be a nice cross-test of that].

Regarding colors and color corrections from V to LSST bandpasses: I took mean reflectance spectra from Bus-DeMeo (http://sbn.psi.edu/pds/resource/busdemeotax.html) and multiplied this by a Kurucz model of the solar spectrum (I don't think was the exact site I got the spectra from, but it should be similar - http://kurucz.harvard.edu/sun.html) to get "observed" mean spectra for each type. These spectra are part of CatSim and are available online - I also just put them http://www.astro.washington.edu/users/lynnej/seds/asteroids.tar (this also contains the kurucz_sun spectra I used to create the reflectance spectra as well as the Harris_V band throughput curve I used to create reference "V" band magnitudes).
I should add that the original reflectance spectra ended at 440 nm, so I have extended the spectra to 300 nm (edge of LSST's u band) by estimating the slope of the reflectance spectra near 440 nm and using this slope to extend the values to 300nm.

The LSST filters I used are the current throughput estimates - available here https://github.com/lsst/throughputs and calculated colors using Sed/Bandpass code available in our sims_photUtils package (https://github.com/lsst/sims_photUtils).

I'll put a piece of code in the repo that will take these inputs and calculate the colors - that way you could change it up and calculate PS colors etc to check against the PS1 reported values. (I don't have a PS1 filter throughput set .. Or you could point me to a PS1 filter throughput set). [code is in: https://github.com/testMOPS/mops/blob/master/calc_colors.py]

The colors I calculated are a little different than the ones you suggested up there Peter - but I'm not sure if much of that comes from using a C/S average? The u band color is quite different though, it may be that my extension in the u band is not very good.

Sed V-u V-g V-r V-i V-z V-y C -1.614 -0.302 0.172 0.291 0.298 0.303 S -1.927 -0.395 0.255 0.455 0.401 0.406

@stevechesley I could treat all asteroids the same; I can also tell you which one was C and which was S (I'm only using these two types). Any preference? (when we do the full model, I think it is easier for me to use the type recorded in the database).

[rhiannonlynne]

BTW, in my Grav S3M NEO model, I have 268897 NEOs. Does this agree with your population? (may be slightly off due to propagation back to 1994 .. we had a few things which OpenOrb didn't like and flung into the sun or a planet, but we didn't track this down very carefully under the assumption that it was probably still pretty close to a reasonable distribution - just looking at the distribution of a/e/i the pre and post propagation orbits looked similar.).

[stevechesley]

The Grav file on the website now has 268511 NEOs. 386 fewer than you have. Maybe @pveres can check the S0 file that PS MOPS is using?

[pveres]

I have 268896 NEO in my database. So, besides one object, I have the same number. Lynne, do you have original designations of these objects? I can compare our datasets.

[rhiannonlynne]

My NEOs are indexed/designated by integers. I don't know if the original mapping we made from the hexadecimal IDs in the PSMOPS model to those integers still exists; however, I think we could make a good guess because I think the files are just 'in order'. Also - I realized that my number of objects is actually the same as yours - I mistakenly included the header line when I "counted" my objects! So we have the same number.

I will work on generating a full S3M set of detections from our database. For now, I have a short analysis of the outputs from the single night, as was generated by MafSSO. I added this to the repo, under the 'n747' directory. I'll start a new issue to deal with the comparison.