apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.83k stars 1.77k forks source link

[Bug] [checkpoint] checkpoint timeout value without effect #5333

Open Xuzhengz opened 1 year ago

Xuzhengz commented 1 year ago

Search before asking

What happened

原因:cdc读取100万的表报错,提示在seatunnel.yaml增加checkpoint超时时长,但是设置后不生效,集群已重启

SeaTunnel Version

2.3.3

SeaTunnel Config

{
  "env": {
    "execution.parallelism": "10",
    "job.mode": "STREAMING",
    "checkpoint.interval": "7000"
  },
  "source": [
    {
      "base-url": "jdbc:mysql://172.16.11.131:3306/test1mysql?remarks=true&useInformationSchema=true&useCursorFetch=true&defaultFetchSize=2048",
      "password": "******",
      "startup.mode": "INITIAL",
      "table-names": [
        "test1mysql.mysql_cdc_demo_source"
      ],
      "result_table_name": "mysql_cdc_demo_source",
      "database-names": [
        "test1mysql"
      ],
      "plugin_name": "MySQL-CDC",
      "connect.max-retries": 3,
      "connect.timeout.ms": 3000,
      "username": "mysql"
    }
  ],
  "transform": [],
  "sink": [
    {
      "password": "******",
      "database": "test1mysql",
      "batch_size": 5000,
      "primary_keys": [
        "pk"
      ],
      "driver": "com.mysql.cj.jdbc.Driver",
      "source_table_name": "mysql_cdc_demo_source",
      "plugin_name": "jdbc",
      "generate_sink_sql": true,
      "enable_upsert": true,
      "user": "mysql",
      "url": "jdbc:mysql://172.16.11.131:3306/test1mysql?remarks=true&useInformationSchema=true&useCursorFetch=true&defaultFetchSize=2048",
      "table": "mysql_cdc_demo_sink",
      "batch_interval_ms": 1000
    }
  ]
}

Running Command

bin/seatunnel.sh -c job.json

Error Exception

Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

f95eaa4601626dc518b087becd799b2

Are you willing to submit PR?

Code of Conduct

davidzollo commented 1 year ago

looking forward to your PR

pppeace commented 1 year ago

遇到相同的问题,目前把 checkpoint 配置去掉可临时解决。请问这个 bug 下个 release 会修复吗?

1151674616 commented 1 year ago

遇到相同的问题,目前把 checkpoint 配置去掉可临时解决。请问这个 bug 下个 release 会修复吗?

请问你是咋去掉的,我2.3.3版本去掉依旧是报错: Please increase checkpoint timeout in the seatunnel.yaml

pppeace commented 1 year ago

遇到相同的问题,目前把 checkpoint 配置去掉可临时解决。请问这个 bug 下个 release 会修复吗?

请问你是咋去掉的,我2.3.3版本去掉依旧是报错: Please increase checkpoint timeout in the seatunnel.yaml

config/seatunnel.yaml 供参考(直接删除了 checkpoint 相关配置):

seatunnel:
  engine:
    history-job-expire-minutes: 1440
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: true
github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

liugddx commented 9 months ago

Is this problem solved?