DUNE-DAQ / drunc

Dune RUN Control (DRUNC) is the run control for the DUNE experiment
1 stars 1 forks source link

Unable to retrieve process logs #236

Closed cc-a closed 1 month ago

cc-a commented 2 months ago

Whilst attempting to retrieve process logs via ProcessManagerDriver.logs I got the below backtrace. Looks like some missing imports in grpc_utils.py. I did have a play around to include the imports and encountered further complaints about protobuff datatypes so there seems to be a wider issue at play.

> Command 'logs' failed on 'process_manager' (response flag 'UNHANDLED_EXCEPTION_THROWN')
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
>     return runner.run(main)
>            ^^^^^^^^^^^^^^^^
>   File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
>     return self._loop.run_until_complete(task)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
>     return future.result()
>            ^^^^^^^^^^^^^^^
>   File "/usr/src/app/main/views.py", line 27, in get_process_logs
>     return [item async for item in pmd.logs(query)]
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/local/lib/python3.12/site-packages/drunc/process_manager/process_manager_driver.py", line 197, in logs
>     async for stream in self.send_command_for_aio(
>   File "/usr/local/lib/python3.12/site-packages/drunc/utils/shell_utils.py", line 222, in send_command_for_aio
>     yield self.handle_response(s, command, outformat)
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/local/lib/python3.12/site-packages/drunc/utils/shell_utils.py", line 175, in handle_response
>     raise DruncServerSideError(error_txt, stack_txt, server_response=dr)
> drunc.exceptions.DruncServerSideError: Stacktrace [bold red]on remote server![/bold red]
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.11/site-packages/drunc/utils/grpc_utils.py", line 96, in unpack_request
>     data = unpack_any(request.data, data_type)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/local/lib/python3.11/site-packages/drunc/utils/grpc_utils.py", line 22, in unpack_any
>     raise UnpackingError(data, format)
> drunc.utils.grpc_utils.UnpackingError: Cannot unpack type_url: "type.googleapis.com/dunedaq.druncschema.ProcessQuery"
> value: "\n&\n$26a6b979-f5cb-47b2-be7d-0a5519191251"
>  to <google._upb._message.Descriptor object at 0x75ed9a20ad90>
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.11/site-packages/drunc/broadcast/server/decorators.py", line 100, in wrap
>     async for a in cmd(obj, request):
>   File "/usr/local/lib/python3.11/site-packages/drunc/authoriser/decorators.py", line 63, in check_token
>     async for a in cmd(obj, request):
>   File "/usr/local/lib/python3.11/site-packages/drunc/utils/grpc_utils.py", line 98, in unpack_request
>     yield Response(
>           ^^^^^^^^
> NameError: name 'Response' is not defined
> 
> 
> NameError: name 'Response' is not defined
>  process_manager -> 6
> 
plasorak commented 2 months ago

Hi Christopher, which version of drunc are you using? The log endpoint seems to be working for me.

cc-a commented 2 months ago

Thanks @plasorak. Managed to resolve this as a mistake on my part but would be nice to fix this up as the errors were a red herring.

plasorak commented 1 month ago

Fixed in https://github.com/DUNE-DAQ/drunc/pull/279