PrioritySchedule's results not always restricted to requested windows

[thusser]

The PriorityScheduler doesn't always restrict the blocks to the requested time windows. I often get 15 seconds and more scheduled outside those windows. This is a problem for us, since the database storing the schedule doesn't accept slots that are not 100% within the requested windows. My current workaround is to make the time constraints smaller than requested. The SequentialScheduler doesn't have the same problem.

I couldn't really find a good minimum working example, with just a few observing blocks in the schedule everything seems fine. So here is an example that loads a list of 23 blocks from file and schedules them. It's just a list of of mostly TESS follow-ups we tried to do two nights ago...

Here is the code (the required blocks.yaml is attached at the end):

from astroplan import AtNightConstraint, Transitioner, PriorityScheduler, Observer, TimeConstraint, Schedule, \
    AirmassConstraint, MoonSeparationConstraint, ObservingBlock, FixedTarget
from astropy.coordinates import SkyCoord, EarthLocation
from astropy.time import Time
import astropy.units as u
import yaml

# load blocks
with open('blocks.yaml', 'r') as f:
    blocks = yaml.safe_load(f)

# create observing blocks
observing_blocks = []
for name, config in blocks.items():
    observing_blocks.append(
        ObservingBlock(
            FixedTarget(SkyCoord(config['ra'] * u.deg, config['dec'] * u.deg, frame='icrs'), name=name),
            config['duration'] * u.second, config['priority'],
            constraints=[
                AirmassConstraint(max=config['max_airmass'], boolean_constraint=False),
                MoonSeparationConstraint(min=config['min_lunar_distance'] * u.deg),
                TimeConstraint(Time(config['start']), Time(config['end']))
            ]
        )
    )

# observer
observer = Observer(EarthLocation.from_geodetic(10. * u.deg, 51. * u.deg))

# only global constraint is the night
constraints = [AtNightConstraint.twilight_nautical()]

# we don't need any transitions
transitioner = Transitioner()

# create scheduler
scheduler = PriorityScheduler(constraints, observer, transitioner=transitioner)

# run scheduler
time_range = Schedule(Time('2021-02-26T14:00:00'), Time('2021-02-27T14:00:00'))
schedule = scheduler(observing_blocks, time_range)

for name, start, end in schedule.to_table().iterrows('target', 'start time (UTC)', 'end time (UTC)'):
    # get block
    block = blocks[name]

    # convert
    block_start, block_end = Time(block['start']), Time(block['end'])
    sched_start, sched_end = Time(start), Time(end)

    # print it
    print(name)
    print('Window:    %s - %s' % (block_start.isot, block_end.isot))
    print('Scheduled: %s - %s' % (sched_start.isot, sched_end.isot))
    if sched_start < block_start or sched_end > block_end:
        print('   SCHEDULED OUTSIDE REQUESTED WINDOW!')
    print('---')

The output of this is:

HIP 24565
Window:    2021-02-26T17:49:50.294 - 2021-02-26T21:49:50.294
Scheduled: 2021-02-26T18:09:20.009 - 2021-02-26T18:11:40.009
---
M42
Window:    2021-02-26T08:09:50.000 - 2021-02-27T08:09:50.000
Scheduled: 2021-02-26T18:28:00.010 - 2021-02-26T18:30:40.010
---
TOI 1676.01
Window:    2021-02-26T19:23:33.700 - 2021-02-26T20:24:39.181
Scheduled: 2021-02-26T19:23:40.012 - 2021-02-26T20:14:40.012
---
TOI 959.01
Window:    2021-02-26T21:28:55.708 - 2021-02-26T22:57:13.764
Scheduled: 2021-02-26T21:29:00.016 - 2021-02-26T22:47:20.016
---
TOI 1274.01
Window:    2021-02-27T01:11:39.252 - 2021-02-27T01:46:50.182
Scheduled: 2021-02-27T01:21:40.025 - 2021-02-27T01:47:00.025
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---
TOI 2138.01
Window:    2021-02-27T02:05:12.480 - 2021-02-27T02:38:50.308
Scheduled: 2021-02-27T02:15:20.027 - 2021-02-27T02:39:00.027
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---
TOI 1829.01
Window:    2021-02-27T03:06:53.686 - 2021-02-27T03:48:25.220
Scheduled: 2021-02-27T03:17:00.029 - 2021-02-27T03:48:40.029
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---
TOI 1252.01
Window:    2021-02-27T04:16:14.102 - 2021-02-27T04:49:17.384
Scheduled: 2021-02-27T04:26:20.032 - 2021-02-27T04:49:20.032
   SCHEDULED OUTSIDE REQUESTED WINDOW!

As you can see, some of the scheduled slots do not fit in the requested time constraints.

Package versions are: astroplan==0.8 astropy==4.2

Any ideas?

blocks.yaml:

HIP 24565:
  dec: 42.75634774
  duration: 123
  end: '2021-02-26T21:49:50.294000Z'
  max_airmass: 2.0
  min_lunar_distance: 15.0
  name: HIP 24565
  priority: 215.25
  ra: 79.05261651
  start: '2021-02-26T17:49:50.294000Z'
M42:
  dec: -5.3911
  duration: 151
  end: '2021-02-27T08:09:50Z'
  max_airmass: 3.0
  min_lunar_distance: 30.0
  name: M42
  priority: 0.0
  ra: 83.8221
  start: '2021-02-26T08:09:50Z'
TOI 1161.01:
  dec: 46.8682472222222
  duration: 4656
  end: '2021-02-27T04:35:52.520000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1161.01
  priority: 38.8
  ra: 286.971375
  start: '2021-02-27T03:08:08.057000Z'
TOI 1165.01:
  dec: 66.3587083333333
  duration: 9493
  end: '2021-02-27T02:25:37.426000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1165.01
  priority: 166.1275
  ra: 232.146625
  start: '2021-02-26T23:37:15.823000Z'
TOI 1252.01:
  dec: 70.7845027777778
  duration: 1375
  end: '2021-02-27T04:49:17.384000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1252.01
  priority: 11.458333333333334
  ra: 266.686083333333
  start: '2021-02-27T04:16:14.102000Z'
TOI 1274.01:
  dec: 65.7474388888889
  duration: 1502
  end: '2021-02-27T01:46:50.182000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1274.01
  priority: 12.516666666666667
  ra: 254.816166666667
  start: '2021-02-27T01:11:39.252000Z'
TOI 1297.01:
  dec: 67.8234361111111
  duration: 16197
  end: '2021-02-27T03:44:24.122000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1297.01
  priority: 283.44750000000005
  ra: 195.821125
  start: '2021-02-26T23:04:18.548000Z'
TOI 1341.01:
  dec: 59.4907694444444
  duration: 2451
  end: '2021-02-27T02:42:39.286000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1341.01
  priority: 20.425
  ra: 301.55925
  start: '2021-02-27T01:51:39.931000Z'
TOI 1425.01:
  dec: 68.9980333333333
  duration: 7594
  end: '2021-02-27T03:17:45.066000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1425.01
  priority: 132.895
  ra: 329.091375
  start: '2021-02-27T01:01:02.445000Z'
TOI 1582.01:
  dec: 64.3563833333333
  duration: 13048
  end: '2021-02-27T00:34:24.430000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1582.01
  priority: 108.73333333333333
  ra: 34.697875
  start: '2021-02-26T20:46:48.145000Z'
TOI 1612.01:
  dec: 85.2333194444445
  duration: 1643
  end: '2021-02-26T18:38:31.823000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1612.01
  priority: 13.691666666666666
  ra: 277.478083333333
  start: '2021-02-26T18:01:00.454000Z'
TOI 1676.01:
  dec: 57.8172083333333
  duration: 3057
  end: '2021-02-26T20:24:39.181000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1676.01
  priority: 25.475
  ra: 65.4695833333333
  start: '2021-02-26T19:23:33.700000Z'
TOI 1780.01:
  dec: 64.9638444444445
  duration: 3752
  end: '2021-02-27T04:44:03.129000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1780.01
  priority: 31.266666666666666
  ra: 165.167208333333
  start: '2021-02-27T03:31:22.896000Z'
TOI 1810.01:
  dec: 44.9153666666667
  duration: 10243
  end: '2021-02-27T00:35:38.663000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1810.01
  priority: 179.2525
  ra: 188.266208333333
  start: '2021-02-26T21:34:46.858000Z'
TOI 1820.01:
  dec: 27.4520027777778
  duration: 15985
  end: '2021-02-27T01:44:19.999000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1820.01
  priority: 279.7375
  ra: 187.686708333333
  start: '2021-02-26T21:07:46.975000Z'
TOI 1829.01:
  dec: 78.7542166666667
  duration: 1883
  end: '2021-02-27T03:48:25.220000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1829.01
  priority: 15.691666666666666
  ra: 290.877416666667
  start: '2021-02-27T03:06:53.686000Z'
TOI 1834.01:
  dec: 31.4019166666667
  duration: 10226
  end: '2021-02-27T04:30:07.386000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1834.01
  priority: 178.955
  ra: 202.29425
  start: '2021-02-27T01:29:33.129000Z'
TOI 1852.01:
  dec: 23.1209361111111
  duration: 14563
  end: '2021-02-27T03:58:15.425000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 1852.01
  priority: 254.85250000000002
  ra: 197.072041666667
  start: '2021-02-26T23:45:24.092000Z'
TOI 2138.01:
  dec: 41.5600194444444
  duration: 1409
  end: '2021-02-27T02:38:50.308000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 2138.01
  priority: 11.741666666666667
  ra: 281.661916666667
  start: '2021-02-27T02:05:12.480000Z'
TOI 2139.01:
  dec: 29.8790416666667
  duration: 2842
  end: '2021-02-27T02:53:49.357000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 2139.01
  priority: 23.683333333333334
  ra: 267.284875
  start: '2021-02-27T01:56:19.055000Z'
TOI 2258.01:
  dec: 62.7909388888889
  duration: 10015
  end: '2021-02-27T02:07:58.626000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 2258.01
  priority: 175.2625
  ra: 268.029541666667
  start: '2021-02-26T23:10:54.926000Z'
TOI 959.01:
  dec: 4.71511666666667
  duration: 4690
  end: '2021-02-26T22:57:13.764000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 959.01
  priority: 39.083333333333336
  ra: 114.012458333333
  start: '2021-02-26T21:28:55.708000Z'
TOI 986.01:
  dec: 10.3438694444444
  duration: 12586
  end: '2021-02-27T00:03:56.569000Z'
  max_airmass: 2.0
  min_lunar_distance: 30.0
  name: TOI 986.01
  priority: 220.25500000000002
  ra: 120.105458333333
  start: '2021-02-26T20:24:01.767000Z'

[wtgee]

Thanks @thusser.

It looks like it might be an issue with the time_resolution parameter that is passed to a scheduler, which has a default of 20 seconds. If you change that, to e.g. one second, it still schedules outside of your time block but it's within about one half second of that (see below).

This is probably an indexing error somewhere in there. I don't have a ton of time right not to try to fix it but that point you in the right direction.

I changed time resolution here. Note that it does take a lot longer to process.

# create scheduler with fine time resolution
scheduler = PriorityScheduler(constraints, observer, transitioner=transitioner, time_resolution=1*u.second)

which gives:

HIP 24565
Window:    2021-02-26T17:49:50.294 - 2021-02-26T21:49:50.294
Scheduled: 2021-02-26T18:09:27.919 - 2021-02-26T18:11:30.919
---
M42
Window:    2021-02-26T08:09:50.000 - 2021-02-27T08:09:50.000
Scheduled: 2021-02-26T18:27:47.913 - 2021-02-26T18:30:18.913
---
TOI 1676.01
Window:    2021-02-26T19:23:33.700 - 2021-02-26T20:24:39.181
Scheduled: 2021-02-26T19:23:33.895 - 2021-02-26T20:14:30.895
---
TOI 959.01
Window:    2021-02-26T21:28:55.708 - 2021-02-26T22:57:13.764
Scheduled: 2021-02-26T21:28:55.854 - 2021-02-26T22:47:05.854
---
TOI 1274.01
Window:    2021-02-27T01:11:39.252 - 2021-02-27T01:46:50.182
Scheduled: 2021-02-27T01:21:48.778 - 2021-02-27T01:46:50.778
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---
TOI 2138.01
Window:    2021-02-27T02:05:12.480 - 2021-02-27T02:38:50.308
Scheduled: 2021-02-27T02:15:21.761 - 2021-02-27T02:38:50.761
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---
TOI 1829.01
Window:    2021-02-27T03:06:53.686 - 2021-02-27T03:48:25.220
Scheduled: 2021-02-27T03:17:02.741 - 2021-02-27T03:48:25.741
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---
TOI 1252.01
Window:    2021-02-27T04:16:14.102 - 2021-02-27T04:49:17.384
Scheduled: 2021-02-27T04:26:22.718 - 2021-02-27T04:49:17.718
   SCHEDULED OUTSIDE REQUESTED WINDOW!
---

[bmorris3]

Thanks both!

Some follow-up questions for @thusser based on @wtgee's post:

• how quickly do you need this to run? You could make the time_resolution parameter even smaller (0.1*u.s) and then you'll get precision on that order, but the runtime will scale with the time resolution. If you're running this once per day then perhaps speed doesn't matter too much, but of course I would understand if you needed to run it on the fly (in response to weather, for example) and you'd like it to be fast
• are the blocks correctly scheduled in terms of order but just slightly too long? It looks like in your examples, the block overflow is occurring on the block end rather than start each time

[bmorris3]

Investigating a bit, I wonder if this is related to these lines: by taking the np.ceil, are we rounding up when we should be rounding down?

Switching that ceil to a floor produces the following (without changing the time_resolution parameter), with passing tests:

HIP 24565
Window:    2021-02-26T17:49:50.294 - 2021-02-26T21:49:50.294
Scheduled: 2021-02-26T18:09:20.009 - 2021-02-26T18:11:20.009
---
M42
Window:    2021-02-26T08:09:50.000 - 2021-02-27T08:09:50.000
Scheduled: 2021-02-26T18:28:00.010 - 2021-02-26T18:30:20.010
---
TOI 1676.01
Window:    2021-02-26T19:23:33.700 - 2021-02-26T20:24:39.181
Scheduled: 2021-02-26T19:23:40.012 - 2021-02-26T20:14:20.012
---
TOI 959.01
Window:    2021-02-26T21:28:55.708 - 2021-02-26T22:57:13.764
Scheduled: 2021-02-26T21:29:00.016 - 2021-02-26T22:47:00.016
---
TOI 1274.01
Window:    2021-02-27T01:11:39.252 - 2021-02-27T01:46:50.182
Scheduled: 2021-02-27T01:21:40.025 - 2021-02-27T01:46:40.025
---
TOI 2138.01
Window:    2021-02-27T02:05:12.480 - 2021-02-27T02:38:50.308
Scheduled: 2021-02-27T02:15:20.027 - 2021-02-27T02:38:40.027
---
TOI 1829.01
Window:    2021-02-27T03:06:53.686 - 2021-02-27T03:48:25.220
Scheduled: 2021-02-27T03:17:00.029 - 2021-02-27T03:48:20.029
---
TOI 1252.01
Window:    2021-02-27T04:16:14.102 - 2021-02-27T04:49:17.384
Scheduled: 2021-02-27T04:26:20.032 - 2021-02-27T04:49:00.032
---

[thusser]

It looks like it might be an issue with the time_resolution parameter that is passed to a scheduler, which has a default of 20 seconds.

Ah, interesting, so at least I know how much smaller I have to make the observing windows for my workaround to work. This is clearly a bug and it would be great if someone would fix it, but that would work for me.

how quickly do you need this to run? You could make the time_resolution parameter even smaller (0.1*u.s) and then you'll get precision on that order, but the runtime will scale with the time resolution. If you're running this once per day then perhaps speed doesn't matter too much, but of course I would understand if you needed to run it on the fly (in response to weather, for example) and you'd like it to be fast

I'm rescheduling quite often: blocks are added during the night, maybe we have to reschedule during the night after we opened late due to clouds, etc. In the future I'd even like to add seeing as a scheduling parameter, in order to skip faint targets or crowded fields. Right now it takes the scheduler a couple of minutes to run wit 30-40 blocks. A longer run is not really an option for us...

are the blocks correctly scheduled in terms of order but just slightly too long? It looks like in your examples, the block overflow is occurring on the block end rather than start each time

Yep, it always seems the block end. The funny thing is that there always is more than enough time at the beginning of the observng window. Look at the last scheduled blocks from the example and compare their start times with the end times of the previous block: they could always start at the beginning of their window.

[wtgee]

I'm rescheduling quite often: blocks are added during the night, maybe we have to reschedule during the night after we opened late due to clouds, etc. In the future I'd even like to add seeing as a scheduling parameter, in order to skip faint targets or crowded fields. Right now it takes the scheduler a couple of minutes to run with 30-40 blocks. A longer run is not really an option for us...

Do you need to schedule the entire night if you are adding/removing during the night? We essentially use a priority dispatch scheduler to answer the question of "what can I observe right now?" and rely on a combination of target priority and constraint weight scoring to give us the "correct" target. It doesn't always work perfectly but that's because we don't have such stringent time windows so we haven't taken the time to make it Always Work™.

Could you schedule in, say, three hour time blocks with a finer time resolution? Your observing windows are all fairly small so that might work.

That said, just getting the code fixed might be the better answer. 😄