search

dagster-io / fake-star-detector

https://github.com/dagster-io/dagster
234 stars 19 forks source link

[Issue] Error while running simpler_model: I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url #4

Open kgolubic opened 1 year ago

kgolubic commented 1 year ago

When the simpler_model gets to around 1000 stargazers, I get the time-out error.

Is there a setting to change the number of retires or to do something else so that it can be run to the end?

There are 1 unused configuration paths:
- models.dbt_project.example
2023-03-20 12:54:08 +0100 - dagster - DEBUG - __ASSET_JOB - 959fd084-49dc-4785-b9ce-4d435f8dc8e7 - stargazers_with_user_info - Completed 900 of 1212 stargazers.
2023-03-20 12:54:20 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
2023-03-20 12:54:27 +0100 - dagster - INFO - __ASSET_JOB - 959fd084-49dc-4785-b9ce-4d435f8dc8e7 - stargazers_with_user_info - 4800 tokens left
2023-03-20 12:55:07 +0100 - dagster.builtin - WARNING - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.dbt_project.example
2023-03-20 12:55:20 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
2023-03-20 12:55:53 +0100 - dagster - DEBUG - __ASSET_JOB - 959fd084-49dc-4785-b9ce-4d435f8dc8e7 - stargazers_with_user_info - Completed 1000 of 1212 stargazers.
2023-03-20 12:56:20 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
2023-03-20 12:56:21 +0100 - dagster.builtin - WARNING - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.dbt_project.example
2023-03-20 12:57:20 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/HerrMuna (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f8e1cc260a0>, 'Connection to api.github.com timed out. (connect timeout=15)'))
2023-03-20 12:57:36 +0100 - dagster.builtin - WARNING - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.dbt_project.example
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/HamletTanyavong (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/rempas (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/timmy-time (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/xxidbr9 (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/xshadowlegendx (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/amerikan (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/vpavicic (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/guzmud (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/asampal (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/ogiste (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/brandonheath (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/57-Wolve (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/sguzman (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/DataBoyTX (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/AlpriElse (Caused by SSLError(OSError(24, 'Too many open files')))
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/mgilbir (Caused by SSLError(OSError(24, 'Too many open files')))
2023-03-20 12:58:20 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
I ran into an error - I will skip this one | Error: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /users/Krazybug (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f8e1cb47cd0>, 'Connection to api.github.com timed out. (connect timeout=15)'))
2023-03-20 12:58:46 +0100 - dagster.builtin - WARNING - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.dbt_project.example

Then there is a large number of Max retries errors and then at one point there are new errors:

Exception while writing logger call to event log: (sqlite3.OperationalError) unable to open database file
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('959fd084-49dc-4785-b9ce-4d435f8dc8e7', '{"__class__": "EventLogEntry", "dagster_event": {"__class__": "DagsterEvent", "event_specific_data": {"__class__": "StepOutputData", "metadata_entrie ... (16019 characters truncated) ... gazers_with_user_info", "timestamp": 1679313767.2596595, "user_message": "Yielded output \\"result\\" of type \\"DataFrame\\". (Type check passed)."}', 'STEP_OUTPUT', '2023-03-20 12:02:47.259660', 'stargazers_with_user_info', None, None)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
Exception while writing logger call to event log: (sqlite3.OperationalError) unable to open database file
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('959fd084-49dc-4785-b9ce-4d435f8dc8e7', '{"__class__": "EventLogEntry", "dagster_event": {"__class__": "DagsterEvent", "event_specific_data": {"__class__": "StepFailureData", "error": {"__cl ... (7691 characters truncated) ... step_key": "stargazers_with_user_info", "timestamp": 1679313767.2749624, "user_message": "Execution of step \\"stargazers_with_user_info\\" failed."}', 'STEP_FAILURE', '2023-03-20 12:02:47.274962', 'stargazers_with_user_info', None, None)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-03-20 13:02:47 +0100 - dagster - ERROR - __ASSET_JOB - 959fd084-49dc-4785-b9ce-4d435f8dc8e7 - 15841 - RUN_FAILURE - Execution of run for "__ASSET_JOB" failed. An exception was thrown during execution.

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('959fd084-49dc-4785-b9ce-4d435f8dc8e7', '{"__class__": "EventLogEntry", "dagster_event": {"__class__": "DagsterEvent", "event_specific_data": {"__class__": "StepFailureData", "error": {"__cl ... (7691 characters truncated) ... step_key": "stargazers_with_user_info", "timestamp": 1679313767.2749624, "user_message": "Execution of step \\"stargazers_with_user_info\\" failed."}', 'STEP_FAILURE', '2023-03-20 12:02:47.274962', 'stargazers_with_user_info', None, None)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Stack Trace:
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/api.py", line 990, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/executor/in_process.py", line 41, in execute
    yield from iter(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/api.py", line 1101, in __iter__
    yield from self.iterator(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 114, in inner_plan_execution_iterator
    for step_event in check.generator(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 358, in dagster_event_sequence_for_step
    yield step_failure_event_from_exc_info(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/objects.py", line 124, in step_failure_event_from_exc_info
    return DagsterEvent.step_failure_event(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 802, in step_failure_event
    return DagsterEvent.from_step(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 413, in from_step
    log_step_event(step_context, event)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 292, in log_step_event
    step_context.log.log_dagster_event(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 405, in log_dagster_event
    self.log(level=level, msg=msg, extra={DAGSTER_META_KEY: dagster_event})
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 420, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 286, in emit
    handler.handle(dagster_record)
  File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 187, in emit
    self._instance.handle_new_event(event)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 1839, in handle_new_event
    self._event_storage.store_event(event)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/storage/event_log/sqlite/sqlite_event_log.py", line 243, in store_event
    conn.execute(insert_event_statement)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)

The above exception was caused by the following exception:
sqlite3.OperationalError: unable to open database file

Stack Trace:
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)

The above exception occurred during handling of the following exception:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('959fd084-49dc-4785-b9ce-4d435f8dc8e7', '{"__class__": "EventLogEntry", "dagster_event": {"__class__": "DagsterEvent", "event_specific_data": {"__class__": "StepOutputData", "metadata_entrie ... (16019 characters truncated) ... gazers_with_user_info", "timestamp": 1679313767.2596595, "user_message": "Yielded output \\"result\\" of type \\"DataFrame\\". (Type check passed)."}', 'STEP_OUTPUT', '2023-03-20 12:02:47.259660', 'stargazers_with_user_info', None, None)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Stack Trace:
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 269, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 384, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event):
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 434, in _type_check_and_store_output
    for output_event in _type_check_output(step_context, step_output_handle, output, version):
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 287, in _type_check_output
    yield DagsterEvent.step_output_event(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 771, in step_output_event
    return DagsterEvent.from_step(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 413, in from_step
    log_step_event(step_context, event)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 292, in log_step_event
    step_context.log.log_dagster_event(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 405, in log_dagster_event
    self.log(level=level, msg=msg, extra={DAGSTER_META_KEY: dagster_event})
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 420, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 286, in emit
    handler.handle(dagster_record)
  File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 187, in emit
    self._instance.handle_new_event(event)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 1839, in handle_new_event
    self._event_storage.store_event(event)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/dagster/_core/storage/event_log/sqlite/sqlite_event_log.py", line 243, in store_event
    conn.execute(insert_event_statement)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)

The above exception was caused by the following exception:
sqlite3.OperationalError: unable to open database file

Stack Trace:
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/kruno/repos/fake-star-detector/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)

2023-03-20 13:03:10 +0100 - dagster.builtin - WARNING - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.dbt_project.example
2023-03-20 13:03:20 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.

image

rachfop commented 1 year ago

I had some success modifying this value, but then I ran into more issue.

yuhan commented 1 year ago

Thanks for raising this issue. I commented a fix here: https://github.com/dagster-io/fake-star-detector/issues/2#issuecomment-1476635797

kgolubic commented 1 year ago

I've tried to switch to Postgres. It still stalls around 1000 stars, and I get the error message Max retries exceeded with URL:

I've switched to the Docker version, and it works so I'll use it for now.