Open chriscomeau79 opened 1 year ago
Hi! Thanks for reporting this. Let me take it back to the team to investigate.
I'm still seeing this problem in 1.4.16
The problem is still here in 1.5.11 with python 3.11. It seems to make the assets page pretty slow to return data about the assets.
Any workaround? Even disabling websocket from the user settings doesn't stop those errors:
i am having the same issue, in my case, Dagit looks like its working totally fine, and i see no visible errors in daemon processes or in the dagit code locations, daemon health pages, etc. I am able to browse the dashboard, see runs, run details, etc.
The last log i have from the dagster webserver process is below.
[32m2024-02-08 03:57:22 -0500[0m - dagster-webserver - [34mINFO[0m - Serving dagster-webserver on http://0.0.0.0:3000 in process 26048
...
[32m2024-02-08 10:17:14 -0500[0m - dagster.code_server - [34mINFO[0m - Shutting down Dagster code server for package etl2.repos.prod on port 64784 in process 30904
WARNING: Invalid HTTP request received.
Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
handle: <Handle _ProactorBasePipeTransport._call_connection_lost(None)>
Traceback (most recent call last):
File "C:\program files\python311\Lib\asyncio\events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "C:\program files\python311\Lib\asyncio\proactor_events.py", line 165, in
_call_connection_lost
self._sock.shutdown(socket.SHUT_RDWR)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote
host
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
handle: <Handle _ProactorBasePipeTransport._call_connection_lost(None)>
Traceback (most recent call last):
File "C:\program files\python311\Lib\asyncio\events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "C:\program files\python311\Lib\asyncio\proactor_events.py", line 165, in
_call_connection_lost
self._sock.shutdown(socket.SHUT_RDWR)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote
host
Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
handle: <Handle _ProactorBasePipeTransport._call_connection_lost(None)>
Traceback (most recent call last):
File "C:\program files\python311\Lib\asyncio\events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "C:\program files\python311\Lib\asyncio\proactor_events.py", line 165, in
_call_connection_lost
self._sock.shutdown(socket.SHUT_RDWR)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote
host
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
The following error message occurs when I try launching a job from the Dagit UI.
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\utils.py", line 125, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\utils.py", line 56, in _fn
result = fn(self, graphene_info, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\schema\roots\mutation.py", line 301, in mutate
return create_execution_params_and_launch_pipeline_exec(graphene_info, executionParams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\schema\roots\mutation.py", line 279, in create_execution_params_and_launch_pipeline_exec
return launch_pipeline_execution(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\execution\launch_execution.py", line 33, in launch_pipeline_execution
return _launch_pipeline_execution(graphene_info, execution_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\execution\launch_execution.py", line 67, in _launch_pipeline_execution
run = do_launch(graphene_info, execution_params, is_reexecuted)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\execution\launch_execution.py", line 50, in do_launch
dagster_run = create_valid_pipeline_run(graphene_info, external_job, execution_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\execution\run_lifecycle.py", line 70, in create_valid_pipeline_run
external_execution_plan = get_external_execution_plan_or_raise(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster_graphql\implementation\external.py", line 100, in get_external_execution_plan_or_raise
return graphene_info.context.get_external_execution_plan(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_core\workspace\context.py", line 244, in get_external_execution_plan
).get_external_execution_plan(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_core\host_representation\code_location.py", line 748, in get_external_execution_plan
execution_plan_snapshot_or_error = sync_get_external_execution_plan_grpc(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_api\snapshot_execution_plan.py", line 52, in sync_get_external_execution_plan_grpc
api_client.execution_plan_snapshot(
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_grpc\client.py", line 231, in execution_plan_snapshot
res = self._query(
^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_grpc\client.py", line 167, in _query
self._raise_grpc_exception(
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_grpc\client.py", line 150, in _raise_grpc_exception
raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B::1%5D:62745: socket is null"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B::1%5D:62745: socket is null {created_time:"2024-02-08T19:02:15.431885+00:00", grpc_status:14}"
>
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_grpc\client.py", line 165, in _query
return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\dagster\_grpc\client.py", line 140, in _get_response
return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\grpc\_channel.py", line 1161, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Python\Environments\Dagster\Lib\site-packages\grpc\_channel.py", line 1004, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python version: 3.11.3 Dagster version: 1.5.6
Anyone figure out a fix for this?
Dagster version
1.3.1
What's the issue?
When running dagit in python 3.11, I noticed the progress updates in the UI were choppy (every 5-10 seconds). Checked the Chrome dev console and I see messages like this every few seconds:
This still happens even if the "disable WebSockets" toggle is enabled.
I tried a few older dagit+dagster versions and the behavior is the same, as long it it's on python 3.11 1.3.1 1.0.0 0.15.9 0.14.20
I switched Dagit to run in a Python 3.10 env and now the run/op progress is updating smoothly.
What did you expect to happen?
No response
How to reproduce?
pip install dagster and dagit in a python 3.11 environment, then run 'dagster dev'
Deployment type
Local
Deployment details
Running inside a conda env with dagster and dagit installed with pip. Same when running 'dagster dev' or 'dagit' locally on Windows 10/11, or running as a service on Ubuntu 22.04
Additional information
Working around this by running dagit and dagster-daemon in a python 3.10 env, then the code locations are in separate python 3.11 envs
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.