talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
425 stars 97 forks source link

Logging Error/ [Erno 22]invalid argument after start of interference #1913

Open Flymaster01 opened 3 weeks ago

Flymaster01 commented 3 weeks ago

Bug description

I have 40 videos of single animal interence. I wanna got my prediction using a trained model trained by me. It worked for 2 month and just out of a sudden i got the following error message.

Expected behaviour

Actual behaviour

Your personal set up

I got folloqong erromessage --- Logging error --- Traceback (most recent call last): File "C:\Users\agfialaa\.conda\envs\sleap\lib\logging\__init__.py", line 1029, in emit self.flush() File "C:\Users\agfialaa\.conda\envs\sleap\lib\logging\__init__.py", line 1009, in flush self.stream.flush() OSError: [Errno 22] Invalid argument Call stack: File "C:\Users\agfialaa\.conda\envs\sleap\Scripts\sleap-track-script.py", line 33, in sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')()) File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 5387, in main f"Auto-selected GPU {gpu_ind} with {mem} MiB of free memory." Message: 'Auto-selected GPU 0 with 12115 MiB of free memory.' Arguments: () 2024-08-19 13:12:12.485810: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-08-19 13:12:13.278267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9593 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:01:00.0, compute capability: 8.6 2024-08-19 13:12:16.532313: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201 Traceback (most recent call last): File "C:\Users\agfialaa\.conda\envs\sleap\Scripts\sleap-track-script.py", line 33, in sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')()) File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 5424, in main labels_pr = predictor.predict(provider) File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 526, in predict self._make_labeled_frames_from_generator(generator, data) File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1583, in _make_labeled_frames_from_generator for ex in generator: File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 485, in _predict_generator flush=True, OSError: [Errno 22] Invalid argument Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='cp1252'> OSError: [Errno 22] Invalid argument Traceback (most recent call last): File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\gui\learning\dialog.py", line 701, in run items_for_inference=items_for_inference, File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\gui\learning\runners.py", line 571, in run_learning_pipeline new_labeled_frame_count = run_gui_inference(inference_task, items_for_inference) File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\gui\learning\runners.py", line 785, in run_gui_inference gui=gui, File "C:\Users\agfialaa\.conda\envs\sleap\lib\site-packages\sleap\gui\learning\runners.py", line 299, in predict_subprocess line = line.decode().rstrip() UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 48: invalid start byte
Logs
# paste relevant logs here, if any

Screenshots

How to reproduce

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
gitttt-1234 commented 3 weeks ago

Hi @Flymaster01,

Could you please try the workaround given here, which addresses the same error?

Please let us know if the suggestions in #1116 work for you! We'll check back in and dig deeper if needed.

Thanks!

Divya