search

ITISFoundation / osparc-simcore

🐼 osparc-simcore simulation framework
https://osparc.io
MIT License
46 stars 27 forks source link

simcore_service_director_v2 observer raises exception due to invalid `stop_duration` during event #6675

Closed pcrespov closed 2 hours ago

pcrespov commented 3 hours ago

I have just noticed this issue in the logs

Task exception was never retrieved
future: <Task finished name='simcore_service_director_v2.modules.dynamic_sidecar.scheduler._core._scheduler.observe_dy-sidecar_a6913f42-3772-58fd-9d01-f56beaafb410' coro=<observing_single_service() done, defined at /home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_observer.py:86> exception=TypeError("unsupported operand type(s) for +=: 'float' and 'NoneType'")>
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_observer.py", line 134, in observing_single_service
    await attempt_pod_removal_and_data_saving(app, scheduler_data)
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_utils.py", line 390, in attempt_pod_removal_and_data_saving
    ).observe(stop_duration)
      ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/prometheus_client/metrics.py", line 650, in observe
    self._sum.inc(amount)
  File "/home/scu/.venv/lib/python3.11/site-packages/prometheus_client/values.py", line 20, in inc
    self._value += amount
TypeError: unsupported operand type(s) for +=: 'float' and 'NoneType'
GitHK commented 2 hours ago