search

bluerobotics / BlueOS

The open source platform for ROV, USV, robotic system operation, development, and expansion.
https://blueos.cloud/docs/
Other
150 stars 73 forks source link

Kraken: Container logs are not live #2267

Closed Williangalvani closed 2 months ago

Williangalvani commented 9 months ago

Happens when looking at container logs in the UI Side-effect from migration to 3.11, apparently.

Williangalvani commented 9 months ago

probably due to the 3.11 change in here: https://docs.python.org/3/library/asyncio-task.html#id12 image

joaoantoniocardoso commented 9 months ago

Yep, Kraken is broken on master, making it impossible to install extensions:

2023-12-22 15:19:49.532 | ERROR    | logging:callHandlers:1706 - Exception in ASGI application

Traceback (most recent call last):

  File "/home/pi/services/kraken/main.py", line 175, in <module>
    loop.run_until_complete(server.serve())
    │    │                  │      └ <function Server.serve at 0xb5aaf7a8>
    │    │                  └ <uvicorn.server.Server object at 0xb551a5f0>
    │    └ <function BaseEventLoop.run_until_complete at 0xb62d4758>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>

  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
    │    └ <function BaseEventLoop.run_forever at 0xb62d4708>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
    │    └ <function BaseEventLoop._run_once at 0xb62d6668>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
    handle._run()
    │      └ <function Handle._run at 0xb6382f28>
    └ <Handle <TaskStepMethWrapper object at 0xb6128f70>()>
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
    │    │            │    │           │    └ <member '_args' of 'Handle' objects>
    │    │            │    │           └ <Handle <TaskStepMethWrapper object at 0xb6128f70>()>
    │    │            │    └ <member '_callback' of 'Handle' objects>
    │    │            └ <Handle <TaskStepMethWrapper object at 0xb6128f70>()>
    │    └ <member '_context' of 'Handle' objects>
    └ <Handle <TaskStepMethWrapper object at 0xb6128f70>()>
> File "/usr/local/lib/python3.11/site-packages/uvicorn-0.13.4-py3.11.egg/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
                   │   │    │      │    │        │    └ <function RequestResponseCycle.send at 0xb54ce848>
                   │   │    │      │    │        └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xb54df9d0>
                   │   │    │      │    └ <function RequestResponseCycle.receive at 0xb54ce898>
                   │   │    │      └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xb54df9d0>
                   │   │    └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
                   │   └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xb54df9d0>
                   └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0xb551a430>
  File "/usr/local/lib/python3.11/site-packages/uvicorn-0.13.4-py3.11.egg/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
                 │    │   │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xb54df9d0>>
                 │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xb54df9d0>>
                 │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
                 │    └ <uvicorn.middleware.message_logger.MessageLoggerMiddleware object at 0xb54cae30>
                 └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0xb551a430>
  File "/usr/local/lib/python3.11/site-packages/uvicorn-0.13.4-py3.11.egg/uvicorn/middleware/message_logger.py", line 65, in __call__
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/uvicorn-0.13.4-py3.11.egg/uvicorn/middleware/message_logger.py", line 61, in __call__
    await self.app(scope, inner_receive, inner_send)
          │    │   │      │              └ <function MessageLoggerMiddleware.__call__.<locals>.inner_send at 0xb60c27a8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <fastapi.applications.FastAPI object at 0xb5512a10>
          └ <uvicorn.middleware.message_logger.MessageLoggerMiddleware object at 0xb54cae30>
  File "/usr/local/lib/python3.11/site-packages/fastapi-0.63.0-py3.11.egg/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <function MessageLoggerMiddleware.__call__.<locals>.inner_send at 0xb60c27a8>
                           │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <function MessageLoggerMiddleware.__call__.<locals>.inner_send at 0xb60c27a8>
          │    │                │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xb5512af0>
          └ <fastapi.applications.FastAPI object at 0xb5512a10>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0xb55fdf28>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <starlette.exceptions.ExceptionMiddleware object at 0xb5512ad0>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xb5512af0>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e348>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <fastapi.routing.APIRouter object at 0xb5512a30>
          └ <starlette.exceptions.ExceptionMiddleware object at 0xb5512ad0>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e348>
          │     │      │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │     └ <function Mount.handle at 0xb5d4a9d8>
          └ <starlette.routing.Mount object at 0xb5516030>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/routing.py", line 376, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e348>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <fastapi.applications.FastAPI object at 0xb5512d70>
          └ <starlette.routing.Mount object at 0xb5516030>
  File "/usr/local/lib/python3.11/site-packages/fastapi-0.63.0-py3.11.egg/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e348>
                           │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e348>
          │    │                │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xb5512eb0>
          └ <fastapi.applications.FastAPI object at 0xb5512d70>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0xb544e398>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <starlette.exceptions.ExceptionMiddleware object at 0xb5512e90>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xb5512eb0>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e3e8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <fastapi.routing.APIRouter object at 0xb5512dd0>
          └ <starlette.exceptions.ExceptionMiddleware object at 0xb5512e90>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e3e8>
          │     │      │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │     └ <function Route.handle at 0xb5d4a668>
          └ <commonwealth.utils.apis.GenericErrorHandlingRoute object at 0xb55085f0>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e3e8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          │    └ <function request_response.<locals>.app at 0xb550c0c8>                                                                                                                                                                                                                                                                                                                                                                                                      "blueos" 15:20 22-Dec-23
          └ <commonwealth.utils.apis.GenericErrorHandlingRoute object at 0xb55085f0>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/routing.py", line 44, in app
    await response(scope, receive, send)
          │        │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xb544e3e8>
          │        │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xb60c2898>
          │        └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 9134), 'cl...
          └ <starlette.responses.StreamingResponse object at 0xb54a6230>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/responses.py", line 228, in __call__
    await run_until_first_complete(
          └ <function run_until_first_complete at 0xb5f53c08>
  File "/usr/local/lib/python3.11/site-packages/starlette-0.13.6-py3.11.egg/starlette/concurrency.py", line 16, in run_until_first_complete
    (done, pending) = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
                            │       │    │                  │       └ 'FIRST_COMPLETED'
                            │       │    │                  └ <module 'asyncio' from '/usr/local/lib/python3.11/asyncio/__init__.py'>
                            │       │    └ [<coroutine object StreamingResponse.stream_response at 0xb54e49a8>, <coroutine object StreamingResponse.listen_for_disconnec...
                            │       └ <function wait at 0xb62bdd98>
                            └ <module 'asyncio' from '/usr/local/lib/python3.11/asyncio/__init__.py'>
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 425, in wait
    raise TypeError("Passing coroutines is forbidden, use tasks explicitly.")

TypeError: Passing coroutines is forbidden, use tasks explicitly.
Williangalvani commented 9 months ago

renamed to track the issue that the container logs are not live with the current code (some lines get buffered and only come through after new ones are pushed...)

patrickelectric commented 2 months ago

Fixed by #2781