smnorris / bcdata

Python and command line tools for quick access to DataBC geo-data available via WFS/WCS.
MIT License
29 stars 7 forks source link

db connection can be unstable #171

Closed smnorris closed 8 months ago

smnorris commented 8 months ago

When loading a larger table over the network, the psycopg2 connection can be dropped when attempting to write to the log table:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/bcdata/database.py", line 69, in execute
    result = curs.execute(sql, params)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/bcdata", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/bcdata/cli.py", line 408, in bc2pg
    out_table = bcdata.bc2pg(
  File "/usr/local/lib/python3.10/dist-packages/bcdata/bc2pg.py", line 229, in bc2pg
    db.execute(
  File "/usr/local/lib/python3.10/dist-packages/bcdata/database.py", line 67, in execute
    with self.conn:
psycopg2.InterfaceError: connection already closed
Error: Process completed with exit code 1.

Cleaning up the connection handling in bcdata.database should fix - sqlalchemy should be able to handle any connection issues.