simonsobs / sorunlib

High level library for running observatory operations using OCS.
BSD 2-Clause "Simplified" License
0 stars 0 forks source link

Second KeyboardInterrupt disrupts `seq.scan` shutdown #166

Open BrianJKoopman opened 1 month ago

BrianJKoopman commented 1 month ago

When aborting, if a user hits interrupt multiple times the second interrupt will disrupt the finally block in seq.scan, which depending on when the second interrupt was sent will result in several SMuRF slots continuing streaming and the telescope continuing scanning.

One solution would be to catch KeyboardInterrupts in this finally block so subsequent interrupts don't abort shutdown.

An example run (run 1397 from satp1) where this secondary interrupt was sent:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/sorunlib/seq.py", line 47, in scan
    monitor_process(acu, 'generate_scan', stop_time)
  File "/usr/local/lib/python3.10/dist-packages/sorunlib/_internal.py", line 172, in monitor_process
    time.sleep(check_interval)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 22, in <module>
  File "/usr/local/lib/python3.10/dist-packages/sorunlib/seq.py", line 51, in scan
    run.smurf.stream('off')
  File "/usr/local/lib/python3.10/dist-packages/sorunlib/smurf.py", line 370, in stream
    resp = smurf.stream.wait()
  File "/usr/local/lib/python3.10/dist-packages/ocs/ocs_client.py", line 26, in wait
    return OCSReply(*client.request('wait', name, timeout=timeout))
  File "/usr/local/lib/python3.10/dist-packages/ocs/client_http.py", line 100, in request
    return self.call(self.agent_addr + '.ops', action, op_name, params, **kw)
  File "/usr/local/lib/python3.10/dist-packages/ocs/client_http.py", line 31, in call
    r = requests.post(self.call_url, data=params)
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt