apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.93k stars 1.79k forks source link

[Bug] [seatunnel-engine] separated-cluster-deployment imap config not effective #7275

Closed Xuzhengz closed 2 months ago

Xuzhengz commented 2 months ago

Search before asking

What happened

According to the official website deployment of separated cluster deployment, after configuring Imap, neither local files nor HDFS files were created in the Imap directory

SeaTunnel Version

2.3.6 and dev

SeaTunnel Config

hazelcast.yaml
hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - localhost
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 2
    hazelcast.max.no.heartbeat.seconds: 180
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100
  map:
    engine*:
      map-store:
        enabled: true
        initial-mode: EAGER
        factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
        properties:
          type: hdfs
          namespace: /opt/module/seatunnel/imap
          clusterName: seatunnel
          storage.type: hdfs
         # fs.defaultFS: hdfs://172.16.11.151:8020
          fs.defaultFS: file:///

hazelcast-master.yaml
hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - 172.16.11.151:5801
          - 172.16.11.151:5802
          - 172.16.11.152:5802
          - 172.16.11.153:5802
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 2
    hazelcast.max.no.heartbeat.seconds: 180
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100

Running Command

bin/seatunnel-cluster.sh -d -r master

Error Exception

imap config not effective

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

image image

Are you willing to submit PR?

Code of Conduct

Xuzhengz commented 2 months ago

cluster mode,imap needs to be configured in hazelcast-master.yaml file