getdozer / dozer

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
https://getdozer.io
GNU Affero General Public License v3.0
1.47k stars 117 forks source link

Aerospike sink doesnt work when aerospike source is used #2479

Open karolisg opened 3 months ago

karolisg commented 3 months ago

Describe the bug

Aerospike sink is storing only one record when aerospike source is used

To Reproduce

Steps to reproduce the behavior:

dozer-config.yaml

app_name: dozer-aero
version: 1
connections:
  - name: aerospike_src
    config: !Aerospike
      namespace: test
      sets:
        - DOZER_USERS_LEG_A
      hosts: localhost:3000

  - name: aerospike_dest
    config: !Aerospike
      namespace: test
      sets:
        - DOZER_USERS_LEG_A_DEST_A
      hosts: localhost:3003
sources:
  - name: DOZER_USERS_LEG_A
    table_name: DOZER_USERS_LEG_A
    connection: aerospike_src
    columns:
      - PK
      - a
      - b
      - c

sinks:
  - name: DOZER_USERS_LEG_A_DEST_A
    config: !Aerospike
      connection: aerospike_dest
      metadata_namespace: test
      tables:
       - !AerospikeSinkTable
         set_name: DOZER_USERS_LEG_A_DEST_A
         namespace: test
         source_table_name: DOZER_USERS_LEG_A

Inserting rows with aql

INSERT INTO test.DOZER_USERS_LEG_A (PK, a, b, c) VALUES (1, "1", "1", "1");
INSERT INTO test.DOZER_USERS_LEG_A (PK, a, b, c) VALUES (2, "2", "2", "2");

... N. Share you error

Number of inserted rows to sink remains 1 even when some rows are inserted to source