toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.1k stars 172 forks source link

Plugin don't work with delete or truncate action. #497

Open laeis opened 9 months ago

laeis commented 9 months ago

PGSync version: 2.3.3 Postgres version: 13.6 Elasticsearch version: 8.7.1 Redis version: latest Python version: 3.7 Problem Description: I created a plugin to filter documents that should not be synced. I use it when moving data from one shard to another because they are both connected to the same opensearch. It works well with Insert and Update. But when I delete the row in postgres, it seems that the plugin doesn't work in this case and also delete document in opensearch. ` from pgsync import plugin class ReadOnlyFilterPlugin(plugin.Plugin): name: str = 'ReadOnlyFilter'

def transform(self, doc: dict, **kwargs) -> dict:
    if 'shard_mapping' in doc and doc['shard_mapping']['read_only'] == False:
        return doc
    else:
        return None

`

Error Message (if any):

ljluestc commented 7 months ago

from pgsync import plugin

class ReadOnlyFilterPlugin(plugin.Plugin):
    name: str = 'ReadOnlyFilter'

    def transform(self, doc: dict, **kwargs) -> dict:
        # Check if it's a delete operation
        if kwargs.get('operation') == 'delete':
            # Implement your logic for delete operation
            # For instance, return None to avoid deleting from Elasticsearch
            return None

        # For non-delete operations, continue with existing logic
        if 'shard_mapping' in doc and doc['shard_mapping']['read_only'] == False:
            return doc
        else:
            return None