Favour ecliptic

[michaelJwilson]

There's less time to get the tiles on the ecliptic given the sun and moon avoidance. Any two tiles being equal, favour ones closer to the ecliptic as they may be harder to get in the future.

Simple implementation that reproduces the previous runs if ecliptic_chase = 0 in the config file. Otherwise, favour tiles close to the ecliptic similar to what is done for the dec_priority. In that case, there were more user knobs available given the rules.yaml, but replicating that for further constraints would lead to an unholy number of groups.

Mixes a little with the dec prioritisation, and the raw exposure times, so not to be applied blindly. Feedback appreciated.

[schlafly]

Do you think of this as a replacement or addition to dec_priority? Currently I imagine dec_priority as a very gentle tie breaker among all tiles at the right LST, after ties have already been broken by things like maintaining a contiguous footprint and trying to build into later passes when possible. The ecliptic and celestial planes intersect in the middle of the NGC and SGC, so this would lead to a tilt in the year-1 footprint. dec_priority has the nice feature that it builds out the HSC survey region first, but otherwise I don't see a reason to prefer dec_priority to ecliptic_priority; I might just replace the former, if ecliptic_priority actually leads to faster survey completion (is this the case?).

[djschlegel]

Ecliptic here is just being used as a proxy for the possible time available for a tile. If we really want to assign higher priority to those tiles that have less visibility time in the future, I'd advocate that we do exactly that instead. It's simple enough to compute. We would compute it separately for dark, gray and bright time. This would automatically take into account the smaller visibility for anything near the equator (thus prioritizing there), near Saturn and Jupiter, and it would even suitably handle the near-the-end-of-the-survey cases.

[michaelJwilson]

I've started this here:

/global/homes/m/mjwilson/repos/desisurvey/py/desisurvey/visibility.py

Pre-computing a look-up table of the hours of visibility / program for each center id and noon-to-noon day based on the same times etc. over which the ephem table is calculated. Should now be good as a v1.

[schlafly]

Ecliptic here is just being used as a proxy for the possible time available for a tile. If we really want to assign higher priority to those tiles that have less visibility time in the future, I'd advocate that we do exactly that instead.

Just thinking out loud here... how crazy would it be to use the annealing strategy used for the Legacy Survey, but instead of assigning tiles to LSTs, to assign tiles to actual nights & times? That sounds potentially expensive, but would fully solve the problem rather than relying on a priority-based heuristic.

[djschlegel]

Ecliptic here is just being used as a proxy for the possible time available for a tile. If we really want to assign higher priority to those tiles that have less visibility time in the future, I'd advocate that we do exactly that instead.

Just thinking out loud here... how crazy would it be to use the annealing strategy used for the Legacy Survey, but instead of assigning tiles to LSTs, to assign tiles to actual nights & times? That sounds potentially expensive, but would fully solve the problem rather than relying on a priority-based heuristic.

Computationally, that shouldn't look too bad. For the imaging surveys, we were annealing solutions for tens of thousands of tiles on a laptop in 15 min. One could easily parallelize to lots of cores with just a little bit of thought to not swap the same tile more than once in the same iteration. But that doesn't avoid the problem that tiles could be scheduled for their last possible observation, so I think we'd still want a penalty function for scheduling a tile late in its list of observable times.

[michaelJwilson]

I need to prioritise other things for much of this week, so in order to save any replication here's a status update: I stripped out the necessary code from David K's surveysim/desisurvey to calculate the visibility. It accounts for

-- Start and end date. -- Fullmoon and monsoon. -- Dawn to Dusk timings on that day. -- Delegation of time to the various programs.
-- Minimum altitude / airmass requirement. -- Collision with the moon, planets etc.

I'd like to give it a final check and there is plenty of scope for optimisation, including the bookkeeping required for trivial parallelisation across CENTERID.

In the meantime, I ran the calculation for the 'potential' one-percent tiles, 160 < ra < 280, -5 < dec < 75, and the dates of 15th April to 15th May. The results are

Dark: visibility-nofullmoon-26-0.pdf

Gray: visibility-nofullmoon-26-1.pdf

Bright: visibility-nofullmoon-26-2.pdf

Let me know if anything jumps out. The code is in visibility.py, which should be wrapped to run as stage between surveyinit and surveysim (using the ephem table provided by the former, calculating initial priorities additional to dec_priorities for the latter).

[dkirkby]

I am catching up now on this PR but missing some context. What problem with the existing algorithm are you trying to fix? You mentioned that tiles on the ecliptic are more constrained, but I don't think simulations of the 5-year survey show this is a problem in practice, i.e., we don't have holes along the ecliptic entering year 5. Have you found evidence to the contrary?

Presumably this is a problem specific to the BRIGHT program? (and perhaps GRAY?) Is it specific to the SV 1% survey?

[dkirkby]

Just thinking out loud here... how crazy would it be to use the annealing strategy used for the Legacy Survey, but instead of assigning tiles to LSTs, to assign tiles to actual nights & times?

The obvious problem is that LST assignments are robust to incorrect assumptions about the actual observing conditions, both during a night and over longer time scales, while datetime assignments are not so would need to be recomputed frequently. What would the advantage of this approach be?

[michaelJwilson]

Presumably this is a problem specific to the BRIGHT program? (and perhaps GRAY?) Is it specific to the SV 1% survey?

Jeremy Tinker has found this for the Bright program, where the pass number along the ecliptic is reduced:

https://desi.lbl.gov/trac/wiki/BrightGalaxyWG/SurveySimulations

This is also apparent in the 1% run. It was less clear this was such a problem for gray in the 1%. Generally, Kyle, Stephen and David thought it'd be a worthwhile addition.

[dkirkby]

Looking at the top plot on that page, the regions with less coverage do not seem to correlate with the ecliptic.

[michaelJwilson]

There was an argument that the bottom left, dec=0 at 12h sat on the ecliptic and was causing the reduced coverage, but I agree there's clearly more going on. This was one reason why David wanted to step away from simply using the ecliptic as a proxy. Even given completely uniform coverage, I think it can also only help the margin.

[schlafly]

The obvious problem is that LST assignments are robust to incorrect assumptions about the actual observing conditions, both during a night and over longer time scales, while datetime assignments are not so would need to be recomputed frequently. What would the advantage of this approach be?

During dark time we do a pretty good job of getting the LSTs we want. During bright time that is much less true, since the moon is often in the way. Agreed that this would require regular date & time reassignments.

[michaelJwilson]

Update on this, as I realised I got caught out by the dec,ra ordering in the moon interpolator.

Five year: visibility-y5-1642-0.pdf visibility-y5-1642-1.pdf visibility-y5-1642-2.pdf

One percent: visibility-op-26-0.pdf visibility-op-26-1.pdf visibility-op-26-2.pdf

Summary: Dark Time visibility remains nicely correlated with declination. This is much less true of both gray and bright, which to first order are better determined by distance from the ecliptic.
Effect is greater still for the one percent survey. I realise the simulator is already accounting for some of this scaling in terms of the moon separation.

I also removed some code that makes it run much faster. It's there if anyone wants it.

[michaelJwilson]

I have a newer version of this in which the survey simulator loads an external priorities file, together with a pre-burning script that generates those priorities consistent with what we have now, but favoring the ecliptic to the same degree. Closing this until I can prove that it genuinely improves the strategy.