search

dagster-io / fake-star-detector

https://github.com/dagster-io/dagster
234 stars 19 forks source link

[Issus]dagster_postgres.utils.DagsterPostgresException: too many retries for DB connection #13

Open lihuacai168 opened 12 months ago

lihuacai168 commented 12 months ago

my config

simpler_model start cmd: dagster dev

db

docker run --name some-postgres \
  -e POSTGRES_PASSWORD=mysecretpassword \
  -p 5432:5432 \
  --ulimit nofile=11136:11136 \
  -d postgres

job

notice: this repo has 10k stars

ops:
  stargazers:
    config:
      repo: metersphere/metersphere
resources:
  github:
    config:
      access_token:
        env: GITHUB_ACCESS_TOKEN

execution:
  config:
    in_process: null

webUI error log

An unexpected exception was thrown. Please file an issue.
dagster_postgres.utils.DagsterPostgresException: too many retries for DB connection

  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_plan.py", line 273, in dagster_event_sequence_for_step
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 481, in core_dagster_event_sequence_for_step
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 536, in _type_check_and_store_output
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 793, in _store_output
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/events/__init__.py", line 943, in asset_materialization
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/events/__init__.py", line 430, in from_step
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/events/__init__.py", line 309, in log_step_event
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/log_manager.py", line 407, in log_dagster_event
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/log_manager.py", line 422, in log
  File "/Users/rikasai/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1624, in _log
  File "/Users/rikasai/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1634, in handle
  File "/Users/rikasai/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
  File "/Users/rikasai/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 968, in handle
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/log_manager.py", line 288, in emit
  File "/Users/rikasai/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 968, in handle
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 206, in emit
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 2105, in handle_new_event
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster_postgres/event_log/event_log.py", line 204, in store_event
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster/_core/storage/event_log/sql_event_log.py", line 366, in store_asset_event_tags
  File "/Users/rikasai/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 135, in __enter__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster_postgres/utils.py", line 165, in create_pg_connection
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster_postgres/utils.py", line 129, in retry_pg_connection_fn

The above exception was caused by the following exception:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "127.0.0.1", port 5432 failed: could not create socket: Too many open files

(Background on this error at: https://sqlalche.me/e/20/e3q8)

  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/dagster_postgres/utils.py", line 117, in retry_pg_connection_fn
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 284, in _do_get
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 615, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect

The above exception was caused by the following exception:
psycopg2.OperationalError: connection to server at "127.0.0.1", port 5432 failed: could not create socket: Too many open files

  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 284, in _do_get
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 615, in connect
  File "/Users/rikasai/.pyenv/versions/3.10.8/envs/fake-star/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect

console log

I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/feiyafei624 (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x129a810c0>: Failed to establish a new connection: [Errno 24] Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/ParkerZsh (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(54, 'Connection reset by peer')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/RegMX (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1299ef790>: Failed to establish a new connection: [Errno 24] Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/pintoke (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1299a5ed0>: Failed to establish a new connection: [Errno 24] Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/ba1q1 (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1299ad480>: Failed to establish a new connection: [Errno 24] Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/codinglin (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1299ef6d0>: Failed to establish a new connection: [Errno 24] Too many open files')))