toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.15k stars 180 forks source link

Incompatibility with timescale #171

Open gabrielpra1 opened 3 years ago

gabrielpra1 commented 3 years ago

Problem Description:

It seems pgsync does not work with TimescaleDB, throwing the following error during bootstrap:

Traceback (most recent call last):,
  File "/usr/local/bin/bootstrap", line 66, in <module>,
    main(),
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in __call__,
    return self.main(*args, **kwargs),
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1062, in main,
    rv = self.invoke(ctx),
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke,
    return ctx.invoke(self.callback, **ctx.params),
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 763, in invoke,
    return __callback(*args, **kwargs),
  File "/usr/local/bin/bootstrap", line 61, in main,
    sync.setup(),
  File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 166, in setup,
    self.teardown(drop_view=False),
  File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 210, in teardown,
    self.drop_triggers(schema=schema, tables=tables),
  File "/usr/local/lib/python3.7/site-packages/pgsync/base.py", line 649, in drop_triggers,
    for table in self.tables(schema):,
  File "/usr/local/lib/python3.7/site-packages/pgsync/base.py", line 181, in tables,
    metadata.reflect(self.__engine),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4710, in reflect,
    Table(name, self, **reflect_opts),
  File "<string>", line 2, in __new__,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/deprecations.py", line 298, in warned,
    return fn(*args, **kwargs),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 600, in __new__,
    metadata._remove_table(name, schema),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__,
    with_traceback=exc_tb,,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_,
    raise exception,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 595, in __new__,
    table._init(name, metadata, *args, **kw),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 675, in _init,
    resolve_fks=resolve_fks,,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 710, in _autoload,
    _extend_on=_extend_on,,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py", line 788, in reflect_table,
    raise exc.NoSuchTableError(table.name),
sqlalchemy.exc.NoSuchTableError: cache_inval_hypertable

This was briefly discussed at https://github.com/toluaina/pgsync/issues/43, but the discussion went another direction.

I also found a similar issue on alembic, also coming from the same underlying problem in SQLAlchemy: https://github.com/sqlalchemy/alembic/issues/733

Is there a workaround for this, or is it not possible to use PGSync with TimescaleDB at the moment?

toluaina commented 3 years ago

Sorry about the delay. Yes I intend to do this. I haven't had the time for new features. This is still in the backlog. Feel free to leave this open. I will take a look at the thread in https://github.com/sqlalchemy/alembic/issues/733