mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Don't show stack traces on Ctrl+c #42

Open martin-loetzsch opened 4 years ago

martin-loetzsch commented 4 years ago

Since a few releases run_pipeline prints various stack traces for KeyboardInterrupt when hitting Ctrl+c:


 ★ 0s
utils: ★ 4.9s
utils / initialize_utils: ★ 4.7s
...
^CProcess system_statistics:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/data-integration/data_integration/execution.py", line 61, in <lambda>
    target=lambda: system_statistics.generate_system_statistics(event_queue), name='system_statistics')
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/data-integration/data_integration/logging/system_statistics.py", line 86, in generate_system_statistics
    time.sleep(period)
KeyboardInterrupt
 Traceback (most recent call last):
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/data-integration/data_integration/execution.py", line 348, in run_pipeline
    _notify_all(event)
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/data-integration/data_integration/execution.py", line 340, in _notify_all
    raise e
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/data-integration/data_integration/execution.py", line 333, in _notify_all
    runlogger.handle_event(event)
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/data-integration/data_integration/logging/run_log.py", line 124, in handle_event
    with mara_db.postgresql.postgres_cursor_context('mara') as cursor:  # type: psycopg2.extensions.cursor
  File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/packages/mara-db/mara_db/postgresql.py", line 18, in postgres_cursor_context
    connection = psycopg2.connect(dbname=db.database, user=db.user, password=db.password,
  File "/Users/mloetzsch/Projects/project-a/mara-example-project-2/.venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
KeyboardInterrupt
Cleaned up open runs/node_runs (run_id = 35)```

While this is not a real problem, it would be nice if we could print something less heavy, e.g. `KeyboardInterrupt`.
jankatins commented 4 years ago

One attempt is now in https://github.com/mara/data-integration/pull/43