toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.1k stars 172 forks source link

Using PGSync with a Heroku managed RDS instance #442

Open LondonAppDev opened 1 year ago

LondonAppDev commented 1 year ago

PGSync version: 2.5.0 Postgres version: 14.7 ~Elasticsearch~ OpenSearch version: 2.5 Redis version: 7.0.10 Python version: 3.11.3

Problem Description:

I am trying to use PGSync with a database that's managed via the Heroku RDS add on.

When I try to run it, I get 'PG_USER "xxxxxxxxx" needs to be superuser or have permission to read, create and destroy replication slots to perform this action.'

This makes sense, because Heroku provisions databases to shared RDS instances, so my user access is limited to the specific database I am given. It's not possible to give my user "superuser" access because it's a shared DB.

Is it possible to use PGSync in this situation?

Thanks in advance.

Error Message (if any):

opensearch-pgsync-1  | Traceback (most recent call last):
opensearch-pgsync-1  |   File "/usr/local/bin/pgsync", line 7, in <module>
opensearch-pgsync-1  |     sync.main()
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
opensearch-pgsync-1  |     return self.main(*args, **kwargs)
opensearch-pgsync-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
opensearch-pgsync-1  |     rv = self.invoke(ctx)
opensearch-pgsync-1  |          ^^^^^^^^^^^^^^^^
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
opensearch-pgsync-1  |     return ctx.invoke(self.callback, **ctx.params)
opensearch-pgsync-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
opensearch-pgsync-1  |     return __callback(*args, **kwargs)
opensearch-pgsync-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/pgsync/sync.py", line 1450, in main
opensearch-pgsync-1  |     sync: Sync = Sync(document, verbose=verbose, **kwargs)
opensearch-pgsync-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/pgsync/singleton.py", line 17, in __call__
opensearch-pgsync-1  |     cls._instances[key] = super(Singleton, cls).__call__(
opensearch-pgsync-1  |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/pgsync/sync.py", line 99, in __init__
opensearch-pgsync-1  |     self.validate(repl_slots=repl_slots)
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/pgsync/sync.py", line 135, in validate
opensearch-pgsync-1  |     self._can_create_replication_slot("_tmp_")
opensearch-pgsync-1  |   File "/usr/local/lib/python3.11/site-packages/pgsync/base.py", line 181, in _can_create_replication_slot
opensearch-pgsync-1  |     raise ReplicationSlotError(
opensearch-pgsync-1  | pgsync.exc.ReplicationSlotError: 'PG_USER "XXXXXXXXX" needs to be superuser or have permission to read, create and destroy replication slots to perform this action.'
opensearch-pgsync-1 exited with code 1
jonjakk commented 1 year ago

For me, increasing max_replication_slots to 3 resolved the issue