apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.01k stars 1.82k forks source link

[Feature][web datasource-postgresql-cdc] support web add pg cdc dadasource #7437

Closed fanxishu closed 1 month ago

fanxishu commented 2 months ago

Search before asking

Description

Added pg cdc plug-in for data synchronization image image env { "job.mode"=STREAMING "job.name"="SeaTunnel_Job" "savemode.execute.location"=CLUSTER } source { Postgres-CDC { "snapshot.split.size"=8096 "snapshot.fetch.size"=1024 "incremental.parallelism"=1 "connect.timeout.ms"=30000 "connect.max-retries"=3 "connection.pool.size"=20 "decoding.plugin.name"=pgoutput "slot.name"=seatunnel "chunk-key.even-distribution.factor.lower-bound"=0.05 "chunk-key.even-distribution.factor.upper-bound"=100 "sample-sharding.threshold"=1000 "startup.mode"=INITIAL "exactly_once"="false" "stop.mode"=NEVER parallelism=1 "result_table_name"=Table14671749401760 catalog { factory=Postgres } database-names=[ ability ] table-names=[ "ability.test.users" ] format=DEFAULT username=ability password="test" base-url="jdbc:postgresql://127.0.0.1:5432/ability?useSSL=false&serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8" server-time-zone=UTC } } transform { } sink { Jdbc { "schema_save_mode"="CREATE_SCHEMA_WHEN_NOT_EXIST" "data_save_mode"="APPEND_DATA" "connection_check_timeout_sec"=30 "batch_size"=1000 "is_exactly_once"="false" "max_commit_attempts"=3 "transaction_timeout_sec"=-1 "max_retries"="3" "auto_commit"="true" "support_upsert_by_query_primary_key_exist"="false" "multi_table_sink_replica"=1 "source_table_name"=Table14671749401760 "generate_sink_sql"=true database=ability table="test2.users" url="jdbc:postgresql://127.0.0.1:5432/ability" user=ability password="test" driver="org.postgresql.Driver" } }

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 month ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.