Refactor of Lccal/Unconex

[rkehoe]

@johnridge, @gigicu01 This is to summarize the needs of the filtering aspect of the lccal/unconex code. In general, the software should be able to accommodate several different types of ROTSE scheduled data:

• sky patrol schedule: This is the data used by NSVS. All observations were taken in pairs throughout the night. The entire visible Northern sky is imaged, and then often a second set of observations occurs, time permitting. These observations may be on the same fields as the first pass, so that observations will either come in closely spaced pairs, or two pairs of observation separated by several hours.
• Orphans schedule: Observations are taken in groups of 5 exposures to maximize the possibility of seeing a GRB in multiple contiguous exposures. A gap of 5 exposure lengths occurs, and then another set of 5 exposures on the original field. Under ideal conditions, these groups always have an odd number of exposures.
• XTE J1118+480 transient schedule: Observations are taken without interruption throughout a night, aside from interruptions from GRB trigger responses.

In terms of how the unconex algorithm should work, here are some thoughts:

1) Adjacent observations that are within a grace period of another Confirm each other. All observations for VSP analysis should be Confirmed. We usually allow a grace period for camera readout and telescope pointing to occur between observations after the prior exposure is over. 2) Confirmed observations should be statistically consistent with at least one of their confirming neighbors to either side in time. A typical threshold is 2 times the error on the difference in their magnitudes. However, this should be overridable with a command line parameter. Once an observation has achieved consistency with one of its confirming observations, it is labelled ‘Consistent’. All observations for VSP analysis must be Consistent. 3) Consistent observations should be paired with one confirming observation and the weighted average of the two measurements calculated, utilizing the weights from the uncertainties on the two observations. 4). The magnitude uncertainty should be calculated from the error propagation of the errors on the two confirming observations.
5). In order to accommodate possibility that some systematic effects are evident in the difference in magnitude between two confirming observations, The difference between these should be compared to the average error of the two observations. If the difference is greater than sort(2)*avg.error, the uncertainty uncertainty on the averaged measurement will be taken as the difference/sqrt(2). 6) Unpaired consistent observations should be retained in the output light curve. 7) There should be no use of the limiting magnitude to throw out observations. Such a parameter can be useful in the relative photometry step, but not in unconex. 8) factor the code such that the ‘filtering’ code is all self-contained in unconex, and the relative photometry code is in lccal, which can call unconex as in the current setup. This will require identifying all the things unconex needs such that it can be given them by lccal, or by a user directly. 9) obsolete the existing ‘standalone’ versions in favor of modular versions along the lines above.

Items 1-7 should be fairly straightward and should be performed first. Items 8 and 9 will be a subsequent development. I suspect that, within unconex, it will be more useful to do the filtering loop first, then loop over the observations for averaging. The averaging is pairing things differently and calculating different things than the filtering loop, and you have to know what observations to remove before you can make intelligent averaging calculations.

Comments?

[gigicu01]

@rkehoe I have a few questions about the statistical part of all this but I think it would be better if we can discuss them in person.

Also, from the explanations John gave yesterday about how Unconex works, it seems to me like Unconex is already using some of the aspects you mentioned. I will have to look through the code though to understand exactly how it works.

After we talk and I understand exactly what the soft should do, and after I learn how to manipulate the data contained in the dat files (I believe I will be able to learn this after I look through Unconex and lccal, otherwise I will ask John if he can spare a few minutes with me to talk about it), I should be ready to go.

[rkehoe]

@gigicu01 @johnridge Adding to item 1) Please convert use of time from match.jd[i] to match.stat[i].obstime.

[gigicu01]

@rkehoe I have managed to access obstime from stat in the match structures and get all the data in there. However, I only changed it to obstime in the conditional where it was checking for consecutive observations. The rest is still using the JD (including the averaging of the epochs), since I can't plot the data with obstime (when using multiple nights) as obstime is only for the time of the day, so all the data points for each night will overlap with those from previous nights. One solution to this if you want to plot using the obstime, would be to use a different plot for each night.

Please let me know what you think.

Also, I have imported the flags from the match strctures and made it so that all the datapoints that had a flag for stauration are completely removed from the analysis (I can easily include the flags for other problems too if necessary).

All of this was done on the previous version of Unconex though. I talked to John and we will probably meet today to combine the new code that he has with mine.

Please, let me know what you think about the obstime problem.

[rkehoe]

@gigicu01 This sounds like a reasonable thing to do with obstime. We generally cannot get much use out of the multi-day light curves because we are looking at variables that have cycles much less than 1 day. So in the multi day plot, it is impossible to see the shape of the light curves or the individual data points.

It may be that unconex should write out both obstime and JD into the output file. The reason is that, when we analyze longer period variables over many days, the multi day plots will be more useful. Certainly, we need the JD for the single_phase and downstream code.

[rkehoe]

@gigicu01 In terms of the removal of saturation entries in the light curve, that sounds great. Please show us tomorrow a light curve before and after just that action, so we can determine if the removal was reasonable. Also, please put the SExtractor flag table in your slides so we can discuss tomorrow and determine if further flags warrange study. I think the deblending on, at least, may be worth a look, since we clearly see evidence of blend/de blend oscillations in some light curves.

[rkehoe]

@gigicu01, @jalasker I believe this Issue has been dealt with in the August PR and I am closing it, since we have just merged PR #20 today.