Human Readable Error Handling

[bh2smith]

This job

  - name: Sync Postgres to Another Postgres Table
    source:
      ref: PG1
      query_string: "SELECT * FROM existing_table"
    destination:
      ref: PG2
      table_name: pg_sync_test_table
      if_exists: append

Produces this ugly-ass error:

.venv➜  dune-sync git:(new-config) ✗ make run
python -m src.main
2024-11-11 18:58:25,916 ERROR dune-sync Invalid SQL query: (psycopg2.errors.UndefinedTable) relation "existing_table" does not exist
LINE 1: EXPLAIN SELECT * FROM existing_table
                              ^

[SQL: EXPLAIN SELECT * FROM existing_table]
(Background on this error at: https://sqlalche.me/e/20/f405)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/bh2smith/Projects/bh2smith/dune-sync/src/main.py", line 16, in <module>
    main()
    ~~~~^^
  File "/Users/bh2smith/Projects/bh2smith/dune-sync/src/main.py", line 8, in main
    config = RuntimeConfig.load_from_yaml((root_path.parent / "config.yaml").absolute())
  File "/Users/bh2smith/Projects/bh2smith/dune-sync/src/config.py", line 155, in load_from_yaml
    source = cls._build_source(job_config["source"], sources)
  File "/Users/bh2smith/Projects/bh2smith/dune-sync/src/config.py", line 183, in _build_source
    return PostgresSource(
        db_url=source.key, query_string=source_config["query_string"]
    )
  File "/Users/bh2smith/Projects/bh2smith/dune-sync/src/sources/postgres.py", line 39, in __init__
    super().__init__()
    ~~~~~~~~~~~~~~~~^^
  File "/Users/bh2smith/Projects/bh2smith/dune-sync/src/interfaces.py", line 18, in __init__
    raise ValueError(f"Config for {self.__class__.__name__} is invalid")
ValueError: Config for PostgresSource is invalid

[mooster531]

With the latest version it produces a different kind of ugly-ass error!

Traceback (most recent call last):
  File "/home/az/workspace/dune-sync/src/main.py", line 67, in <module>
    asyncio.run(main())
  File "/home/az/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/az/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/az/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/az/workspace/dune-sync/src/main.py", line 58, in main
    config = RuntimeConfig.load_from_yaml(args.config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/az/workspace/dune-sync/src/config.py", line 194, in load_from_yaml
    for db_ref in data.get("data_sources", []):
                  ^^^^^^^^
AttributeError: 'list' object has no attribute 'get'

[mooster531]

Ignore last comment, I foolishly copied and pasted the entire example into the config.yaml, instead of just taking it as a job config.

Anyway the issue is addressed in #85 Running the program with this config:

data_sources:
  - name: Dune
    type: dune
    key: ${DUNE_API_KEY}
  - name: PG
    type: postgres
    key: ${DB_URL}

jobs:
  - name: Sync Postgres to Another Postgres Table
    source:
      ref: PG1
      query_string: "SELECT * FROM existing_table"
    destination:
      ref: PG2
      table_name: pg_sync_test_table
      if_exists: append

now crashes properly and informatively:

Fatal: no datasource with `name` = "PG1" defined in config

[mooster531]

I'd keep this issue open for other instances of ugly tracebacks instead of friendly error messages that we're bound to encounter later on.