search

AllenNeuralDynamics / dynamic-foraging-task

Bonsai/Harp workflow for Dynamic Foraging with Python GUI for visualization and control
MIT License
5 stars 4 forks source link

FATAL:ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine #413

Open alexpiet opened 4 months ago

alexpiet commented 4 months ago

Unclear what is happening here. Creating an issue to assess frequency

FATAL:ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine | Foraging.py | exception_hook | line 3668 | ephys1

11:55:21 AM:INFO:Foraging:Foraging.py:log_git_hash:line 3580:Python version: 3.9.19 (main, Mar 21 2024, 17:21:27) [MSC v.1916 64 bit (AMD64)] 11:55:21 AM:INFO:Foraging:Foraging.py:log_git_hash:line 3596:Current git commit branch, hash: main, 3b1b41c58ce82ddd51af4a933079f4715931b292 11:55:21 AM:WARNING:Foraging:Foraging.py:log_git_hash:line 3606:local repository is clean ... 11:55:43 AM:INFO:Foraging:Foraging.py:_restartlogging:line 712:Restarting logging 11:55:43 AM:INFO:rigcontrol:rigcontrol.py:msg_handler:line 54:['/loggerstarted', 'D:\dynamic-foraging\temporaryvideo\2024-04-24_11-55-43\raw.harp', '127.0.0.1', 4002] 11:57:18 AM:INFO:Foraging:Foraging.py:_StopCurrentSession:line 2849:Stopping current trials 11:57:19 AM:ERROR:Foraging:Foraging.py:exception_hook:line 3668:FATAL ERROR: Traceback (most recent call last): File "C:\Users\svc_aind_ephys\AppData\Roaming\Python\Python39\site-packages\pyOSC3\OSC3.py", line 2833, in _transmitWithTimeout tmp = self.socket.send(data[sent:]) ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

XX-Yin commented 4 months ago

It may be caused by directly closing the console.

alexpiet commented 2 months ago

Happened twice on ephys1. @KanghoonJ Do you know what is happening here?

10:18:21 AM:INFO:Foraging:Foraging.py:_StopCurrentSession:line 3339:Stopping current trials
10:18:23 AM:ERROR:Foraging:Foraging.py:exception_hook:line 4180:FATAL ERROR: 
Traceback (most recent call last):
  File "C:\Users\svc_aind_ephys\.conda\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2833, in _transmitWithTimeout
    tmp = self.socket.send(data[sent:])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Foraging.py", line 2062, in closeEvent
    self._stop_camera()
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Foraging.py", line 3265, in _stop_camera
    self.Camera_dialog._StartCamera()
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Dialogs.py", line 1327, in _StartCamera
    self.MainWindow.Channel.CameraControl(int(2))
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\rigcontrol.py", line 157, in CameraControl
    self.send("/cameracontrol", value)
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\rigcontrol.py", line 60, in send
    return self.client.sendOSC(message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\svc_aind_ephys\.conda\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2867, in sendOSC
    txOk = self._transmitMsgWithTimeout(msg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\svc_aind_ephys\.conda\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2858, in _transmitMsgWithTimeout
    if self._transmitWithTimeout(len_big_endian) and self._transmitWithTimeout(binary):
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\svc_aind_ephys\.conda\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2841, in _transmitWithTimeout
    if e[0] == errno.ECONNRESET:
       ~^^^
TypeError: 'ConnectionAbortedError' object is not subscriptable
KanghoonJ commented 2 months ago

It seemed to happen when Bonsai workflow is closed first.

KanghoonJ commented 2 months ago

What is a proper order of closing the task (Foraging GUI, Bonsai, and console windows)? Do we document this somewhere?

alexpiet commented 2 months ago

I would do the following:

  1. Stop Task/Save
  2. Stop Bonsai workflow, then close the workflow (if using with the editor open, which I strongly discourage)
  3. Close the GUI window
  4. Close the console window
alexpiet commented 1 month ago

happened again this week 1 FATAL:ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine Foraging.py exception_hook line 4407 ephys1

alexpiet commented 1 month ago

happened again last week 2 | FATAL:ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine | Foraging.py | exception_hook | line 4424 | 2B, ephys3

@XX-Yin Can you describe what happened when this error happened?

06:13:27 PM:INFO:Foraging:Foraging.py:_Save:line 2282:Saving current session, ForceSave=False
06:13:27 PM:INFO:Foraging:Foraging.py:_StopCurrentSession:line 3459:Stopping current trials
06:13:30 PM:INFO:Foraging:Foraging.py:_Save:line 2282:Saving current session, ForceSave=1
06:13:31 PM:INFO:Foraging:Foraging.py:_Save:line 2293:saving without weight or extra water
06:13:31 PM:INFO:Foraging:Foraging.py:_GetWaterCalibration:line 971:Loaded Water Calibration
06:13:31 PM:INFO:Foraging:Foraging.py:_NewSession:line 3388:New Session pressed
06:13:31 PM:ERROR:Foraging:Foraging.py:exception_hook:line 4424:FATAL ERROR: 
Traceback (most recent call last):
  File "C:\Users\svc_aind_ephys\Anaconda3\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2833, in _transmitWithTimeout
    tmp = self.socket.send(data[sent:])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Foraging.py", line 1432, in _ForceSave
    self._Save(ForceSave=1)
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Foraging.py", line 2462, in _Save
    self._NewSession()
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Foraging.py", line 3403, in _NewSession
    self._stop_camera()
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Foraging.py", line 3373, in _stop_camera
    self.Camera_dialog._StartCamera()
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\Dialogs.py", line 1328, in _StartCamera
    self.MainWindow.Channel.CameraControl(int(2))
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\rigcontrol.py", line 163, in CameraControl
    self.send("/cameracontrol", value)
  File "C:\Users\svc_aind_ephys\Documents\GitHub\dynamic-foraging-task\src\foraging_gui\rigcontrol.py", line 60, in send
    return self.client.sendOSC(message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\svc_aind_ephys\Anaconda3\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2867, in sendOSC
    txOk = self._transmitMsgWithTimeout(msg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\svc_aind_ephys\Anaconda3\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2858, in _transmitMsgWithTimeout
    if self._transmitWithTimeout(len_big_endian) and self._transmitWithTimeout(binary):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\svc_aind_ephys\Anaconda3\envs\Foraging\Lib\site-packages\pyOSC3\OSC3.py", line 2841, in _transmitWithTimeout
    if e[0] == errno.ECONNRESET:
       ~^^^
TypeError: 'ConnectionAbortedError' object is not subscriptable

06:13:52 PM:INFO:Foraging:Foraging.py:_Save:line 2282:Saving current session, ForceSave=False
06:13:52 PM:INFO:Foraging:Foraging.py:_StopCurrentSession:line 3459:Stopping current trials
06:14:27 PM:INFO:Foraging:Foraging.py:_Save:line 2282:Saving current session, ForceSave=1
06:14:28 PM:INFO:Foraging:Foraging.py:_Save:line 2293:saving without weight or extra water
06:14:28 PM:INFO:Foraging:Foraging.py:_GetWaterCalibration:line 971:Loaded Water Calibration
06:14:28 PM:INFO:Foraging:Foraging.py:_NewSession:line 3388:New Session pressed
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: ^C
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4437:BONSAI: 
06:14:46 PM:INFO:Foraging:Foraging.py:log_subprocess_output:line 4439:BONSAI logging terminating
XX-Yin commented 1 month ago

@alexpiet good catch! This was happening on last Friday due to the disk memory is full and then the bonsai was broken. I did not clean the disk. One improvement we can do to prevent this from happening again is to display a warning message and stop the session when disk space is low.

XX-Yin commented 1 month ago

I remember the problem in ephys1 was also related to low memory?