ValueError(f"no moves found between {b_prev} and {b_next}, very unusual!") with SATP3 policy

[ykyohei]

This error does not happen every day, but I see this error with SATP3 policy. If I change sun avoidance rule, 'min_angle' from 49 to 41, there is no error.

Script https://github.com/simonsobs/scheduler-scripts/blob/satp3/satp3/TSATScheduler.py Using latest scheduler today 67a556a0e94393106448f1d7d2282b0e87d2f130

from schedlib.policies import SATP3Policy
import datetime as dt

t0 = dt.datetime(2024, 5, 2, 6, 0, 0, tzinfo=dt.timezone.utc)
t1 = t0 + dt.timedelta(days=1)

master_file = 'master_files/E60_A40_S2023-12-01_F2025-01-01_D-10_-40_L0.86_6.86_12.86_18.86_T65.00_M045_S045.txt'

policy = SATP3Policy.from_defaults(
    master_file = master_file,
    az_speed = 0.5,
    az_accel = 0.25,
    disable_hwp = False,
    apply_boresight_rot = False,
)

seqs = policy.build_schedule(t0, t1)

Error log

2024-05-07 14:29:20,269 [INFO] applying sun avoidance rule: {'min_angle': 49, 'min_sun_time': 1980}
2024-05-07 14:29:20,591 [INFO] planning calibration scans...
2024-05-07 14:29:20,592 [INFO] applying calibration policy - round-robin - to resolve calibration target conflicts
2024-05-07 14:29:20,761 [INFO] -> after calibration policy: []
2024-05-07 14:29:20,762 [INFO] applying min duration rule: {'min_duration': 600}
2024-05-07 14:29:20,762 [INFO] ================ pass 1 ================
2024-05-07 14:29:20,762 [INFO] step 1: planning pre-session ops
2024-05-07 14:29:20,763 [INFO] step 2: planning calibration scans
2024-05-07 14:29:20,786 [INFO] step 3: planning cmb ops
2024-05-07 14:29:20,809 [INFO] step 4: planning post-session ops
2024-05-07 14:29:20,810 [INFO] not enough time for post-session operations, trimming...
2024-05-07 14:29:20,810 [INFO] ================ pass 2 ================
2024-05-07 14:29:20,810 [INFO] step 1: planning pre-session ops
2024-05-07 14:29:20,810 [INFO] step 2: planning calibration scans
2024-05-07 14:29:20,831 [INFO] step 3: planning cmb ops
2024-05-07 14:29:20,854 [INFO] step 4: planning post-session ops
2024-05-07 14:29:20,855 [INFO] ================ pass 3 ================
2024-05-07 14:29:20,855 [INFO] step 1: planning pre-session ops
2024-05-07 14:29:20,855 [INFO] step 2: planning calibration scans
2024-05-07 14:29:20,874 [INFO] step 3: planning cmb ops
2024-05-07 14:29:20,897 [INFO] step 4: planning post-session ops
2024-05-07 14:29:20,898 [WARNING] round_trip: ir did not converge after 3 passes
2024-05-07 14:29:20,898 [INFO] ================ lowering ================
2024-05-07 14:29:20,898 [INFO] step 1: planning pre-session ops
2024-05-07 14:29:20,898 [INFO] step 2: planning calibration scans
2024-05-07 14:29:20,918 [INFO] step 3: planning cmb ops
2024-05-07 14:29:20,942 [INFO] step 4: planning post-session ops
2024-05-07 14:29:20,942 [INFO] ================ solve moves ================
2024-05-07 14:29:20,942 [INFO] step 1: solve sun-safe moves
Traceback (most recent call last):
  File "/home/ykyohei/scheduler-scripts/satp3/TSATScheduler.py", line 24, in <module>
    seqs = policy.build_schedule(t0, t1)
  File "/home/ykyohei/scheduler/src/schedlib/policies/sat.py", line 663, in build_schedule
    ir = self.seq2cmd(seqs, t0, t1, state)
  File "/home/ykyohei/scheduler/src/schedlib/policies/sat.py", line 610, in seq2cmd
    ops = build_op.apply(seq, t0, t1, state, self.operations)
  File "/home/ykyohei/scheduler/src/schedlib/policies/stages/build_op.py", line 131, in apply
    ir = PlanMoves(**self.plan_moves).apply(ir)
  File "/home/ykyohei/scheduler/src/schedlib/policies/stages/build_op.py", line 746, in apply
    moves = recur(pairs)
  File "/home/ykyohei/scheduler/src/schedlib/policies/stages/build_op.py", line 725, in recur
    moves_rest = recur(cdr)
  File "/home/ykyohei/scheduler/src/schedlib/policies/stages/build_op.py", line 725, in recur
    moves_rest = recur(cdr)
  File "/home/ykyohei/scheduler/src/schedlib/policies/stages/build_op.py", line 725, in recur
    moves_rest = recur(cdr)
  [Previous line repeated 13 more times]
  File "/home/ykyohei/scheduler/src/schedlib/policies/stages/build_op.py", line 742, in recur
    raise ValueError(f"no moves found between {b_prev} and {b_next}, very unusual!")
ValueError: no moves found between pre_block       (pre_bloc) az =  111.49: 24-05-02 15:13:01 -> 24-05-02 15:26:02 and Constant elevat (in_block) az =  111.49: 24-05-02 15:26:02 -> 24-05-02 16:26:00, very unusual!

[kmharrington]

I found another one using this config and the satp1 gen_schedule script. Definitely still an issue

# yaml loads iso format automatically into datetimes
t0: 2024-07-09T20:00:00+00:00
t1: 2024-07-10T20:00:00+00:00

elevation: 60
boresight: 0
az_speed: 0.5
az_accel: 1.0

hwp_dir: True
stow_at_end: True
run_relock: False

[mhasself]

This is substantially addressed in #76 -- if similar fails are seen, post them.