apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.1k stars 172 forks source link

[BUG] doris cluster create failed be and cn pod CrashLoopBackOff #7214

Closed JashBook closed 4 months ago

JashBook commented 8 months ago

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. install addon
    helm install doris addons/doris
  2. create cluster
    helm install doris-cluster  addons/doris-cluster
  3. See error
    ➜  ~ kubectl get cluster
    NAME            CLUSTER-DEFINITION   VERSION       TERMINATION-POLICY   STATUS     AGE
    doris-cluster   doris                doris-2.0.3   Delete               Abnormal   22m
    ➜  ~ 
    ➜  ~ kubectl get pod -l app.kubernetes.io/instance=doris-cluster
    NAME                       READY   STATUS             RESTARTS        AGE
    doris-cluster-doris-be-0   0/1     CrashLoopBackOff   3 (16s ago)     22m
    doris-cluster-doris-be-1   0/1     CrashLoopBackOff   3 (15s ago)     22m
    doris-cluster-doris-be-2   0/1     Completed          3 (5m56s ago)   22m
    doris-cluster-doris-cn-0   0/1     CrashLoopBackOff   3 (23s ago)     22m
    doris-cluster-doris-cn-1   0/1     CrashLoopBackOff   3 (29s ago)     22m
    doris-cluster-doris-cn-2   0/1     CrashLoopBackOff   3 (29s ago)     22m
    doris-cluster-doris-fe-0   3/3     Running            0               22m
    doris-cluster-doris-fe-1   3/3     Running            0               22m
    doris-cluster-doris-fe-2   3/3     Running            0               22m

logs pod doris-cluster-doris-be-0

kubectl logs doris-cluster-doris-be-0 --previous 
2024-02-04T03:59:16+00:00 [Warn] [Entrypoint]: BUILD_TYPE k8s
2024-02-04T03:59:16+00:00 [Note] [Entrypoint]: MASTER_FE_IP 
2024-02-04T03:59:16+00:00 [Note] [Entrypoint]: CURRENT_BE_IP 
2024-02-04T03:59:16+00:00 [Note] [Entrypoint]: CURRENT_BE_PORT 
2024-02-04T03:59:16+00:00 [Note] [Entrypoint]: PRIORITY_NETWORKS ...0/24
2024-02-04T03:59:16+00:00 [Note] [Entrypoint]: BE is not register. retry.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/apache-doris/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
I0204 03:59:34.608037   185 daemon.cpp:432]  version doris-2.0.3-rc06(AVX2) RELEASE (build git://VM-16-12-centos@37d31a5550aa6fb40ba5b6696385c7dcd3dbc931)
Built on Sun, 10 Dec 2023 02:35:34 UTC by VM-16-12-centos
I0204 03:59:34.620327   185 mem_info.cpp:463] Physical Memory: 2.00 GB, Mem Limit: 1.60 GB, origin config value: 80%, System Mem Available Min Reserve: 204.80 MB, Vm Min Free KBytes: 66.00 MB, Vm Overcommit Memory: 1
I0204 03:59:34.622553   185 daemon.cpp:440] Cpu Info:
  Model: Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz
  Cores: 4
  Max Possible Cores: 4
  L1 Cache: 48.00 KB (Line: 64.00 B)
  L2 Cache: 1.25 MB (Line: 64.00 B)
  L3 Cache: 48.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    avx
    avx2
  Numa Nodes: 1
  Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 |
I0204 03:59:34.622586   185 daemon.cpp:441] Disk Info: 
  Num disks 2: vda, vdb
I0204 03:59:34.622589   185 daemon.cpp:442] Physical Memory: 2.00 GB
Memory Limt: 1.60 GB
CGroup Info: Process CGroup Info: memory.limit_in_bytes=2147483648, cpu cfs limits: [INVALID_ARGUMENT]quota <= period
I0204 03:59:34.623342   185 backend_options.cpp:101] priority cidrs: 10.96.0.0/12
I0204 03:59:34.623411   185 backend_options.cpp:129] skip ip not belonged to priority networks: 127.0.0.1
I0204 03:59:34.623416   185 backend_options.cpp:129] skip ip not belonged to priority networks: 192.168.36.34
I0204 03:59:34.623418   185 backend_options.cpp:129] skip ip not belonged to priority networks: ::1
I0204 03:59:34.623420   185 backend_options.cpp:129] skip ip not belonged to priority networks: fe80::d4ac:77ff:fe53:776f
F0204 03:59:34.623422   185 backend_options.cpp:133] fail to find one valid address, exit.
*** Check failure stack trace: ***
    @     0x55f5654d9219  google::LogMessageFatal::~LogMessageFatal()
    @     0x55f55dc31455  doris::BackendOptions::analyze_localhost()
    @     0x55f55dc30a02  doris::BackendOptions::init()
    @     0x55f55d149769  main
    @     0x7fbe3f62bd0a  __libc_start_main
    @     0x55f55d14802a  _start
    @              (nil)  (unknown)
*** Query id: 0-0 ***
*** tablet id: 0 ***
*** Aborted at 1707019174 (unix time) try "date -d @1707019174" if you are using GNU date ***
*** Current BE git commitID: 37d31a5 ***
*** SIGABRT unknown detail explain (@0xb9) received by PID 185 (TID 185 OR 0x7fbe3f477e00) from PID 185; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417
 1# 0x00007FBE3F640D60 in /lib/x86_64-linux-gnu/libc.so.6
 2# raise in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x000055F5654DF1F9 in /opt/apache-doris/be/lib/doris_be
 5# google::LogMessage::SendToLog() in /opt/apache-doris/be/lib/doris_be
 6# google::LogMessage::Flush() in /opt/apache-doris/be/lib/doris_be
 7# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris/be/lib/doris_be
 8# doris::BackendOptions::analyze_localhost(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool&, std::vector<doris::CIDR, std::allocator<doris::CIDR> >*, std::vector<doris::InetAddress, std::allocator<doris::InetAddress> >*) at /root/src/doris-2.0/be/src/service/backend_options.cpp:143
 9# doris::BackendOptions::init() at /root/src/doris-2.0/be/src/service/backend_options.cpp:59
10# main at /root/src/doris-2.0/be/src/service/doris_main.cpp:458
11# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
12# _start in /opt/apache-doris/be/lib/doris_be

2024-02-04T03:59:37+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T03:59:57+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:00:17+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:00:37+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:00:57+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:01:17+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:01:38+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:01:58+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:02:18+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:02:38+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:02:58+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:03:18+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:03:38+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:03:58+00:00 [Note] [Entrypoint]: BE is not register. retry.

logs doris-cluster-doris-cn-0

kubectl logs doris-cluster-doris-cn-0 --previous 
2024-02-04T03:59:08+00:00 [Warn] [Entrypoint]: BUILD_TYPE k8s
2024-02-04T03:59:08+00:00 [Note] [Entrypoint]: MASTER_FE_IP 
2024-02-04T03:59:08+00:00 [Note] [Entrypoint]: CURRENT_BE_IP 
2024-02-04T03:59:08+00:00 [Note] [Entrypoint]: CURRENT_BE_PORT 
2024-02-04T03:59:08+00:00 [Note] [Entrypoint]: PRIORITY_NETWORKS ...0/24
2024-02-04T03:59:08+00:00 [Note] [Entrypoint]: BE is not register. retry.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/apache-doris/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
I0204 03:59:24.679215   185 daemon.cpp:432]  version doris-2.0.3-rc06(AVX2) RELEASE (build git://VM-16-12-centos@37d31a5550aa6fb40ba5b6696385c7dcd3dbc931)
Built on Sun, 10 Dec 2023 02:35:34 UTC by VM-16-12-centos
I0204 03:59:24.691677   185 mem_info.cpp:463] Physical Memory: 2.00 GB, Mem Limit: 1.60 GB, origin config value: 80%, System Mem Available Min Reserve: 204.80 MB, Vm Min Free KBytes: 66.00 MB, Vm Overcommit Memory: 1
I0204 03:59:24.693856   185 daemon.cpp:440] Cpu Info:
  Model: Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz
  Cores: 4
  Max Possible Cores: 4
  L1 Cache: 48.00 KB (Line: 64.00 B)
  L2 Cache: 1.25 MB (Line: 64.00 B)
  L3 Cache: 48.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    avx
    avx2
  Numa Nodes: 1
  Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 |
I0204 03:59:24.693892   185 daemon.cpp:441] Disk Info: 
  Num disks 2: vda, vdb
I0204 03:59:24.693895   185 daemon.cpp:442] Physical Memory: 2.00 GB
Memory Limt: 1.60 GB
CGroup Info: Process CGroup Info: memory.limit_in_bytes=2147483648, cpu cfs limits: [INVALID_ARGUMENT]quota <= period
I0204 03:59:24.694545   185 backend_options.cpp:101] priority cidrs: 10.96.0.0/12
I0204 03:59:24.694602   185 backend_options.cpp:129] skip ip not belonged to priority networks: 127.0.0.1
I0204 03:59:24.694607   185 backend_options.cpp:129] skip ip not belonged to priority networks: 192.168.36.33
I0204 03:59:24.694608   185 backend_options.cpp:129] skip ip not belonged to priority networks: ::1
I0204 03:59:24.694610   185 backend_options.cpp:129] skip ip not belonged to priority networks: fe80::fc15:b2ff:fe5c:3597
F0204 03:59:24.694612   185 backend_options.cpp:133] fail to find one valid address, exit.
*** Check failure stack trace: ***
    @     0x556852962219  google::LogMessageFatal::~LogMessageFatal()
    @     0x55684b0ba455  doris::BackendOptions::analyze_localhost()
    @     0x55684b0b9a02  doris::BackendOptions::init()
    @     0x55684a5d2769  main
    @     0x7f1e675fcd0a  __libc_start_main
    @     0x55684a5d102a  _start
    @              (nil)  (unknown)
*** Query id: 0-0 ***
*** tablet id: 0 ***
*** Aborted at 1707019164 (unix time) try "date -d @1707019164" if you are using GNU date ***
*** Current BE git commitID: 37d31a5 ***
*** SIGABRT unknown detail explain (@0xb9) received by PID 185 (TID 185 OR 0x7f1e67448e00) from PID 185; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417
 1# 0x00007F1E67611D60 in /lib/x86_64-linux-gnu/libc.so.6
 2# raise in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x00005568529681F9 in /opt/apache-doris/be/lib/doris_be
 5# google::LogMessage::SendToLog() in /opt/apache-doris/be/lib/doris_be
 6# google::LogMessage::Flush() in /opt/apache-doris/be/lib/doris_be
 7# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris/be/lib/doris_be
 8# doris::BackendOptions::analyze_localhost(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool&, std::vector<doris::CIDR, std::allocator<doris::CIDR> >*, std::vector<doris::InetAddress, std::allocator<doris::InetAddress> >*) at /root/src/doris-2.0/be/src/service/backend_options.cpp:143
 9# doris::BackendOptions::init() at /root/src/doris-2.0/be/src/service/backend_options.cpp:59
10# main at /root/src/doris-2.0/be/src/service/doris_main.cpp:458
11# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
12# _start in /opt/apache-doris/be/lib/doris_be

2024-02-04T03:59:29+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T03:59:49+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:00:09+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:00:29+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:00:50+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:01:10+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:01:30+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:01:50+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:02:10+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:02:30+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:02:50+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:03:10+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:03:31+00:00 [Note] [Entrypoint]: BE is not register. retry.
2024-02-04T04:03:51+00:00 [Note] [Entrypoint]: BE is not register. retry.

logs doris-cluster-doris-fe-0

kubectl logs doris-cluster-doris-fe-0 doris-fe
2024-02-04T03:42:44+00:00 [Warn] [Entrypoint]: BUILD_TYPE k8s
=====
<?xml version="1.0" encoding="utf-8"?>

<!-- Auto Generated. DO NOT MODIFY IT! -->
<Configuration status="info" packages="org.apache.doris.common">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">      <PatternLayout charset="UTF-8">
        <Pattern>%d{yyyy-MM-dd HH:mm:ss,SSS} %p (%t|%tid) [%C{1}.%M():%L] %m%n</Pattern>
      </PatternLayout>
    </Console>    <RollingFile name="Sys" fileName="/opt/apache-doris/fe/log/fe.log" filePattern="/opt/apache-doris/fe/log/fe.log.%d{yyyyMMdd}-%i" immediateFlush="true">
      <PatternLayout charset="UTF-8">
        <Pattern>%d{yyyy-MM-dd HH:mm:ss,SSS} %p (%t|%tid) [%C{1}.%M():%L] %m%n</Pattern>
      </PatternLayout>
      <Policies>
        <TimeBasedTriggeringPolicy/>
        <SizeBasedTriggeringPolicy size="1024MB"/>
      </Policies>
      <DefaultRolloverStrategy max="10" fileIndex="min">
        <Delete basePath="/opt/apache-doris/fe/log/" maxDepth="1">
          <IfFileName glob="fe.log.*" />
          <IfLastModified age="7d" />
        </Delete>
      </DefaultRolloverStrategy>
    </RollingFile>
    <RollingFile name="SysWF" fileName="/opt/apache-doris/fe/log/fe.warn.log" filePattern="/opt/apache-doris/fe/log/fe.warn.log.%d{yyyyMMdd}-%i" immediateFlush="true">
      <PatternLayout charset="UTF-8">
        <Pattern>%d{yyyy-MM-dd HH:mm:ss,SSS} %p (%t|%tid) [%C{1}.%M():%L] %m%n</Pattern>
      </PatternLayout>
      <Policies>
        <TimeBasedTriggeringPolicy/>
        <SizeBasedTriggeringPolicy size="1024MB"/>
      </Policies>
      <DefaultRolloverStrategy max="10" fileIndex="min">
        <Delete basePath="/opt/apache-doris/fe/log/" maxDepth="1">
          <IfFileName glob="fe.warn.log.*" />
          <IfLastModified age="7d" />
        </Delete>
      </DefaultRolloverStrategy>
    </RollingFile>
    <RollingFile name="Auditfile" fileName="/opt/apache-doris/fe/log/fe.audit.log" filePattern="/opt/apache-doris/fe/log/fe.audit.log.%d{yyyyMMdd}-%i">
      <PatternLayout charset="UTF-8">
        <Pattern>%d{yyyy-MM-dd HH:mm:ss,SSS} [%c{1}] %m%n</Pattern>
      </PatternLayout>
      <Policies>
        <TimeBasedTriggeringPolicy/>
        <SizeBasedTriggeringPolicy size="1024MB"/>
      </Policies>
      <DefaultRolloverStrategy max="90" fileIndex="min">
        <Delete basePath="/opt/apache-doris/fe/log/" maxDepth="1">
          <IfFileName glob="fe.audit.log.*" />
          <IfLastModified age="30d" />
        </Delete>
      </DefaultRolloverStrategy>
    </RollingFile>
  </Appenders>
  <Loggers>
    <Root level="INFO" includeLocation="true">
      <AppenderRef ref="Sys"/>
      <AppenderRef ref="SysWF" level="WARN"/>
      <AppenderRef ref="Console"/>

    </Root>
    <Logger name="audit" level="ERROR" additivity="false">
      <AppenderRef ref="Auditfile"/>
    </Logger>
    <Logger name='audit.slow_query' level='INFO'/><Logger name='audit.query' level='INFO'/><Logger name='audit.load' level='INFO'/><Logger name='audit.stream_load' level='INFO'/>
  </Loggers>
</Configuration>
=====
==============================
2024-02-04 03:42:46,100 INFO (main|1) [DorisFE.start():142] Doris FE starting...
2024-02-04 03:42:46,111 INFO (main|1) [FrontendOptions.initAddrUsingFqdn():160] Use FQDN init local addr, FQDN: doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local, IP: 192.168.36.18
2024-02-04 03:42:46,420 INFO (main|1) [ConsistencyChecker.initWorkTime():106] consistency checker will work from 15:00 to 15:00
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris/fe/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/fe/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/fe/lib/hive-catalog-shade-1.0.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-02-04 03:42:47,485 INFO (main|1) [PrivTable.addEntry():89] add priv entry: Node_priv Admin_priv 
2024-02-04 03:42:47,486 INFO (main|1) [PrivTable.addEntry():89] add priv entry: Admin_priv 
2024-02-04 03:42:47,517 INFO (main|1) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:information_schema, priv: Select_priv 
2024-02-04 03:42:47,517 INFO (main|1) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:mysql, priv: Select_priv 
2024-02-04 03:42:47,517 INFO (main|1) [PrivTable.addEntry():89] add priv entry: origWorkloadGroup:normal, priv:Usage_priv 
2024-02-04 03:42:47,518 INFO (main|1) [Auth.createUserInternal():484] finished to create user: 'root'@'%', is replay: true
2024-02-04 03:42:47,518 INFO (main|1) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:information_schema, priv: Select_priv 
2024-02-04 03:42:47,518 INFO (main|1) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:mysql, priv: Select_priv 
2024-02-04 03:42:47,519 INFO (main|1) [PrivTable.addEntry():89] add priv entry: origWorkloadGroup:normal, priv:Usage_priv 
2024-02-04 03:42:47,519 INFO (main|1) [Auth.createUserInternal():484] finished to create user: 'admin'@'%', is replay: true
2024-02-04 03:42:47,719 INFO (main|1) [Env.getSelfHostPort():1228] get self node: HostInfo{host='doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}
2024-02-04 03:42:47,722 INFO (main|1) [DeployManager.initEnvVariables():168] get deploy env: FE_SERVICE, FE_OBSERVER_SERVICE, BE_SERVICE, BROKER_SERVICE, CN_SERVICE
2024-02-04 03:42:47,722 INFO (main|1) [DeployManager.initEnvVariables():173] Electable service group is found
2024-02-04 03:42:47,722 INFO (main|1) [DeployManager.initEnvVariables():187] Backend service group is found
2024-02-04 03:42:47,722 INFO (main|1) [DeployManager.initEnvVariables():201] Cn service group is found
2024-02-04 03:42:47,723 INFO (main|1) [DeployManager.initEnvVariables():206] get electableFeServiceGroup: doris-cluster-doris-fe-headless, observerFeServiceGroup: , backendServiceGroup: doris-cluster-doris-be-headless brokerServiceGroup: , cnServiceGroup: doris-cluster-doris-cn-headless
2024-02-04 03:42:47,723 INFO (main|1) [K8sDeployManager.initEnvVariables():110] use namespace: default
2024-02-04 03:42:47,723 INFO (main|1) [K8sDeployManager.initEnvVariables():118] use domainLTD: svc.cluster.local
[INFO] Env name of: {} is: {}ELECTABLEFE_STATEFULSET
2024-02-04 03:42:47,724 INFO (main|1) [K8sDeployManager.initEnvVariables():132] use statefulSetName: ELECTABLE, doris-cluster-doris-fe
[INFO] Env name of: {} is: {}BACKENDBE_STATEFULSET
2024-02-04 03:42:47,724 INFO (main|1) [K8sDeployManager.initEnvVariables():132] use statefulSetName: BACKEND, doris-cluster-doris-be
[INFO] Env name of: {} is: {}BACKEND_CNCN_STATEFULSET
2024-02-04 03:42:47,724 INFO (main|1) [K8sDeployManager.initEnvVariables():132] use statefulSetName: BACKEND_CN, doris-cluster-doris-cn
2024-02-04 03:42:47,724 INFO (main|1) [DeployManager.getHelperNodes():286] get init num of fe from env: 3
log4j:WARN No appenders could be found for logger (io.fabric8.kubernetes.client.Config).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2024-02-04 03:42:49,809 INFO (main|1) [DeployManager.getHelperNodes():319] get electable fe host from remote: [HostInfo{host='doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}, HostInfo{host='doris-cluster-doris-fe-1.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}, HostInfo{host='doris-cluster-doris-fe-2.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}]
2024-02-04 03:42:49,810 INFO (main|1) [DeployManager.getHelperNodes():325] sorted fe host list: [HostInfo{host='doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}, HostInfo{host='doris-cluster-doris-fe-1.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}, HostInfo{host='doris-cluster-doris-fe-2.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}]
2024-02-04 03:42:49,810 INFO (main|1) [Env.getHelperNodes():1282] get helper nodes: [HostInfo{host='doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local', port=9010}]
2024-02-04 03:42:49,820 INFO (main|1) [Env.getClusterIdAndRole():993] very first time to start this node. role: FOLLOWER, node name: fe_1574497c_7bc7_4483_8f18_b9d02be94586
2024-02-04 03:42:49,821 INFO (main|1) [Env.getClusterIdAndRole():1035] add self frontend: name: fe_1574497c_7bc7_4483_8f18_b9d02be94586, role: FOLLOWER, doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local:9010, is alive: true
2024-02-04 03:42:49,821 INFO (main|1) [Env.getClusterIdAndRole():1153] finished to get cluster id: 1101346986, isElectable: true, role: FOLLOWER and node name: fe_1574497c_7bc7_4483_8f18_b9d02be94586
2024-02-04 03:42:49,882 INFO (main|1) [Env.loadImage():1721] image does not exist: /opt/apache-doris/fe/doris-meta/image/image.0
2024-02-04 03:42:55,601 WARN (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [Env.notifyNewFETypeTransfer():2393] notify new FE type transfer: UNKNOWN
[2024-02-04 03:42:55] notify new FE type transfer: UNKNOWN
2024-02-04 03:42:55,608 WARN (RepNode fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|84) [Env.notifyNewFETypeTransfer():2393] notify new FE type transfer: MASTER
[2024-02-04 03:42:55] notify new FE type transfer: MASTER
2024-02-04 03:43:00,608 WARN (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [Env.notifyNewFETypeTransfer():2393] notify new FE type transfer: MASTER
[2024-02-04 03:43:00] notify new FE type transfer: MASTER
2024-02-04 03:43:00,612 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [BDBJEJournal.open():366] the very first time to open bdb, dbname is 1
2024-02-04 03:43:00,618 INFO (stateListener|96) [Env$4.runOneCycle():2416] begin to transfer FE type from INIT to UNKNOWN
2024-02-04 03:43:00,619 INFO (stateListener|96) [Env$4.runOneCycle():2503] finished to transfer FE type to UNKNOWN
2024-02-04 03:43:00,619 INFO (stateListener|96) [Env$4.runOneCycle():2416] begin to transfer FE type from UNKNOWN to MASTER
2024-02-04 03:43:00,620 INFO (stateListener|96) [BDBHA.fencing():78] start fencing, epoch number is 1
2024-02-04 03:43:00,622 INFO (stateListener|96) [Env.transferToMaster():1343] finish replay in 0 msec
2024-02-04 03:43:00,658 INFO (stateListener|96) [Env.initLowerCaseTableNames():1580] Finish initializing lower_case_table_names, value is 0
2024-02-04 03:43:00,688 INFO (stateListener|96) [PolicyMgr.createDefaultStoragePolicy():113] Create default storage success.
2024-02-04 03:43:00,690 INFO (stateListener|96) [Env.transferToMaster():1392] logMasterInfo:MasterInfo: host=doris-cluster-doris-fe-0.doris-cluster-doris-fe-headless.default.svc.cluster.local httpPort=8030 rpcPort=9020
2024-02-04 03:43:00,690 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:mysql, priv: Select_priv 
2024-02-04 03:43:00,690 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:information_schema, priv: Select_priv 
2024-02-04 03:43:00,690 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: origWorkloadGroup:normal, priv:Usage_priv 
2024-02-04 03:43:00,691 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: Admin_priv 
2024-02-04 03:43:00,691 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:mysql, priv: Select_priv 
2024-02-04 03:43:00,691 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: database privilege.ctl: internal, db: default_cluster:information_schema, priv: Select_priv 
2024-02-04 03:43:00,691 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: origWorkloadGroup:normal, priv:Usage_priv 
2024-02-04 03:43:00,691 INFO (stateListener|96) [PrivTable.addEntry():89] add priv entry: Node_priv Admin_priv 
2024-02-04 03:43:00,692 INFO (stateListener|96) [Env.startMasterOnlyDaemonThreads():1453] checkpointer thread started. thread id is 97
2024-02-04 03:43:00,704 INFO (stateListener|96) [Env.startMasterOnlyDaemonThreads():1489] deploy manager deployManager start
2024-02-04 03:43:00,718 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [Env.waitForReady():938] wait catalog to be ready. FE type: UNKNOWN. is ready: false, counter: 1
2024-02-04 03:43:00,721 INFO (stateListener|96) [K8sDeployManager.startListenerInternal():142] Start listen statefulSets change event.
2024-02-04 03:43:00,724 WARN (Thread-57|103) [InternalSchemaInitializer.run():66] FE is not ready
2024-02-04 03:43:00,724 WARN (mtmv-job-cleaner-scheduler-1-thread-1|104) [MTMVJobManager.lambda$start$0():104] only master can run MTMVJob
2024-02-04 03:43:00,799 INFO (stateListener|96) [Env.checkLowerCaseTableNames():1602] lower_case_table_names is 0
[2024-02-04 03:43:00] master finished to replay journal, can write now.
2024-02-04 03:43:00,799 INFO (stateListener|96) [Env.transferToMaster():1414] master finished to replay journal, can write now.
2024-02-04 03:43:00,807 INFO (colocate group clone checker|101) [ColocateTableCheckerAndBalancer.matchGroups():571] finished to check tablets. unhealth/total/added/in_sched/not_ready: 0/0/0/0/0, cost: 0 ms
2024-02-04 03:43:00,809 INFO (tablet checker|30) [TabletChecker.checkTablets():331] finished to check tablets. unhealth/total/added/in_sched/not_ready: 0/0/0/0/0, cost: 7 ms
2024-02-04 03:43:00,812 INFO (stateListener|96) [Env$4.runOneCycle():2503] finished to transfer FE type to MASTER
2024-02-04 03:43:00,812 INFO (stateListener|96) [Env$4.runOneCycle():2416] begin to transfer FE type from MASTER to MASTER
2024-02-04 03:43:00,818 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [Env.waitForReady():932] catalog is ready. FE type: MASTER
2024-02-04 03:43:00,883 INFO (Automatic Analyzer|44) [StatisticsCollector.runAfterCatalogReady():49] Stats table not available, skip
2024-02-04 03:43:00,884 INFO (stream_load_record_manager|32) [StreamLoadRecordMgr.runAfterCatalogReady():337] finished to pull stream load records of all backends. record size: 0, cost: 0 ms
2024-02-04 03:43:00,884 INFO (single_tablet_load_recorder|33) [SingleTabletLoadRecorderMgr.runAfterCatalogReady():53] Remove expired load tablet record successfully.
2024-02-04 03:43:00,885 INFO (binlog-gcer|45) [BinlogManager.gc():348] begin gc binlog
2024-02-04 03:43:00,885 INFO (binlog-gcer|45) [BinlogManager.gc():359] gc binlog, dbBinlogMap is null
2024-02-04 03:43:00,885 INFO (binlog-gcer|45) [BinlogGcer.runAfterCatalogReady():61] no gc binlog
2024-02-04 03:43:00,883 WARN (Statistics Table Cleaner|42) [StatisticsCleaner.init():118] Failed to init stats cleaner
java.lang.RuntimeException: Table: `internal.default_cluster:__internal_schema.column_statistics` not exists
    at org.apache.doris.statistics.util.StatisticsUtil.findTable(StatisticsUtil.java:370) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.statistics.StatisticsCleaner.init(StatisticsCleaner.java:109) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.statistics.StatisticsCleaner.clear(StatisticsCleaner.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.statistics.StatisticsCleaner.runAfterCatalogReady(StatisticsCleaner.java:74) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-02-04 03:43:00,897 INFO (leaderCheckpointer|97) [Checkpoint.doCheckpoint():89] Http server is not ready.
2024-02-04 03:43:00,906 INFO (tablet stat mgr|27) [TabletStatMgr.runAfterCatalogReady():132] finished to update index row num of all databases. cost: 0 ms
2024-02-04 03:43:01,201 INFO (deployManager|47) [BDBHA.addUnReadyElectableNode():222] set electable group size override to 1, total follower count: 2, add unready node: fe_5ef9a56e_422a_4702_bcf7_556aaf12f2ec
2024-02-04 03:43:01,202 INFO (deployManager|47) [Env.addFrontend():2610] add frontend: name: fe_5ef9a56e_422a_4702_bcf7_556aaf12f2ec, role: FOLLOWER, doris-cluster-doris-fe-1.doris-cluster-doris-fe-headless.default.svc.cluster.local:9010, is alive: false
2024-02-04 03:43:01,282 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add ELECTABLE node: doris-cluster-doris-fe-1.doris-cluster-doris-fe-headless.default.svc.cluster.local:9010
2024-02-04 03:43:01,287 INFO (deployManager|47) [BDBHA.addUnReadyElectableNode():222] set electable group size override to 1, total follower count: 3, add unready node: fe_4f30aee4_a55d_4420_88d8_a1ca59b51d12
2024-02-04 03:43:01,287 INFO (deployManager|47) [Env.addFrontend():2610] add frontend: name: fe_4f30aee4_a55d_4420_88d8_a1ca59b51d12, role: FOLLOWER, doris-cluster-doris-fe-2.doris-cluster-doris-fe-headless.default.svc.cluster.local:9010, is alive: false
2024-02-04 03:43:01,292 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add ELECTABLE node: doris-cluster-doris-fe-2.doris-cluster-doris-fe-headless.default.svc.cluster.local:9010
2024-02-04 03:43:01,404 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=10002, host=doris-cluster-doris-be-0.doris-cluster-doris-be-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 03:43:01,405 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND node: doris-cluster-doris-be-0.doris-cluster-doris-be-headless.default.svc.cluster.local:9050
2024-02-04 03:43:01,481 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=10003, host=doris-cluster-doris-be-1.doris-cluster-doris-be-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 03:43:01,482 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND node: doris-cluster-doris-be-1.doris-cluster-doris-be-headless.default.svc.cluster.local:9050
2024-02-04 03:43:01,485 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=10004, host=doris-cluster-doris-be-2.doris-cluster-doris-be-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 03:43:01,485 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND node: doris-cluster-doris-be-2.doris-cluster-doris-be-headless.default.svc.cluster.local:9050
2024-02-04 03:43:01,581 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=10005, host=doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 03:43:01,581 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND_CN node: doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 03:43:01,586 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=10006, host=doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 03:43:01,586 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND_CN node: doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 03:43:01,588 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=10007, host=doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 03:43:01,588 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND_CN node: doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 03:43:01,693 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [FeServer.start():78] thrift server started.
2024-02-04 03:43:01,910 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 10002 with slots num: 0
2024-02-04 03:43:01,910 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 10003 with slots num: 0
2024-02-04 03:43:01,910 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 10004 with slots num: 0
2024-02-04 03:43:01,910 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 10005 with slots num: 0
2024-02-04 03:43:01,910 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 10006 with slots num: 0
2024-02-04 03:43:01,910 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 10007 with slots num: 0
2024-02-04 03:43:02,002 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [RestartApplicationListener.onApplicationStartingEvent():93] Restart disabled due to System property 'spring.devtools.restart.enabled' being set to false
2024-02-04 03:43:02,089 INFO (-1046302571-pool-18-thread-1|124) [K8sDeployManager$1.eventReceived():389] Watch event received ADDED: doris-cluster-doris-be: 3

        @@@@@@                                                                
        @@@@@@@@                                                              
        @@@@@@@@@                                                             
     @@@  @@@@@@@  @           @@@@@        @@@@      @@@@@     @@     @@@@   
     @@@@@  @@@@  @@@@         @@   @@@   @@    @@@   @@   @@   @@   @@       
     @@@@@@@    @@@@@@@        @@    @@  @@       @   @@   @@   @@   @@@      
     @@@@@@@  @@@@@@@@@        @@     @  @@       @   @@ @@     @@       @@@  
     @@@@@  @@@@@@@@@@         @@   @@@   @@    @@@   @@  @@    @@        @@  
     @@@  @@@@@@@@@@           @@@@@        @@@@      @@    @@  @@    @@@@    
        @@@@@@@@@@                                                            
        @@@@@@@@                                                              
        @@@@@@@                                                               

2024-02-04 03:43:02,915 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [StartupInfoLogger.logStarting():55] Starting DorisFE v1.2-SNAPSHOT using Java 1.8.0_342 on doris-cluster-doris-fe-0 with PID 100 (/opt/apache-doris/fe/lib/doris-fe.jar started by root in /)
2024-02-04 03:43:02,917 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [SpringApplication.logStartupProfileInfo():631] No active profile set, falling back to 1 default profile: "default"
2024-02-04 03:43:03,003 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [DeferredLog.logTo():255] For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'
2024-02-04 03:43:04,383 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [RepositoryConfigurationDelegate.registerRepositoriesIn():132] Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
2024-02-04 03:43:04,404 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [RepositoryConfigurationDelegate.registerRepositoriesIn():201] Finished Spring Data repository scanning in 15 ms. Found 0 LDAP repository interfaces.
2024-02-04 03:43:05,585 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [JettyServletWebServerFactory.getWebServer():166] Server initialized with port: 8030
2024-02-04 03:43:05,621 INFO (UNKNOWN fe_1574497c_7bc7_4483_8f18_b9d02be94586(-1)|1) [ServletWebServerApplicationContext.prepareWebApplicationContext():292] Root WebApplicationContext: initialization completed in 2616 ms
2024-02-04 03:43:06,092 WARN (heartbeat-mgr-pool-5|143) [HeartbeatMgr$BackendHeartbeatHandler.call():266] backend heartbeat got exception
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:255) ~[libthrift-0.16.0.jar:0.16.0]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
    at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
    at org.apache.thrift.transport.TSocket.open(TSocket.java:250) ~[libthrift-0.16.0.jar:0.16.0]
    ... 13 more
...
2024-02-04 04:08:17,915 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10004, beHost: doris-cluster-doris-be-2.doris-cluster-doris-be-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:17,915 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 16173, beHost: doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:17,915 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 16174, beHost: doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:17,915 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 16175, beHost: doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:20,826 INFO (colocate group clone checker|101) [ColocateTableCheckerAndBalancer.matchGroups():571] finished to check tablets. unhealth/total/added/in_sched/not_ready: 0/0/0/0/0, cost: 0 ms
2024-02-04 04:08:20,909 INFO (tablet checker|30) [TabletChecker.checkTablets():331] finished to check tablets. unhealth/total/added/in_sched/not_ready: 0/0/0/0/0, cost: 0 ms
2024-02-04 04:08:21,532 INFO (Thread-57|103) [InternalCatalog.createDb():420] create database[default_cluster:__internal_schema] which already exists
2024-02-04 04:08:21,533 INFO (Thread-57|103) [Database.checkDataSizeQuota():335] database[default_cluster:__internal_schema] data quota: left bytes: 1024.000 TB / total: 1024.000 TB
2024-02-04 04:08:21,533 INFO (Thread-57|103) [Database.checkReplicaQuota():346] database[default_cluster:__internal_schema] replica quota: left number: 1073741824 / total: 1073741824
2024-02-04 04:08:21,533 WARN (Thread-57|103) [InternalSchemaInitializer.run():75] Statistics storage initiated failed, will try again later
org.apache.doris.common.DdlException: errCode = 2, detailMessage = replication num should be less than the number of available backends. replication num is 1, available backend num is 0
    at org.apache.doris.system.SystemInfoService.selectBackendIdsForReplicaCreation(SystemInfoService.java:549) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.InternalCatalog.createTablets(InternalCatalog.java:2766) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.InternalCatalog.createPartitionWithIndices(InternalCatalog.java:1853) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.InternalCatalog.createOlapTable(InternalCatalog.java:2430) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.InternalCatalog.createTable(InternalCatalog.java:1102) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.catalog.InternalSchemaInitializer.createTbl(InternalSchemaInitializer.java:128) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.catalog.InternalSchemaInitializer.run(InternalSchemaInitializer.java:73) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-02-04 04:08:21,556 INFO (deployManager|47) [SystemInfoService.dropBackend():274] finished to drop Backend [id=16173, host=doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}]
2024-02-04 04:08:21,556 INFO (deployManager|47) [DeployManager.dealDropLocal():531] Finished to drop BACKEND node: doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 04:08:21,557 INFO (deployManager|47) [SystemInfoService.dropBackend():274] finished to drop Backend [id=16174, host=doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}]
2024-02-04 04:08:21,558 INFO (deployManager|47) [DeployManager.dealDropLocal():531] Finished to drop BACKEND node: doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 04:08:21,559 INFO (deployManager|47) [SystemInfoService.dropBackend():274] finished to drop Backend [id=16175, host=doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}]
2024-02-04 04:08:21,559 INFO (deployManager|47) [DeployManager.dealDropLocal():531] Finished to drop BACKEND node: doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 04:08:21,578 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=16212, host=doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 04:08:21,578 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND_CN node: doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 04:08:21,579 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=16213, host=doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 04:08:21,579 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND_CN node: doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 04:08:21,580 INFO (deployManager|47) [SystemInfoService.addBackend():224] finished to add Backend [id=16214, host=doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local, heartbeatPort=9050, alive=false, lastStartTime=\N, tags: {location=default}] 
2024-02-04 04:08:21,580 INFO (deployManager|47) [DeployManager.dealAddRemote():562] Finished to add BACKEND_CN node: doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local:9050
2024-02-04 04:08:22,383 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():211] delete non exist backend: 16173
2024-02-04 04:08:22,384 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():211] delete non exist backend: 16174
2024-02-04 04:08:22,384 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():211] delete non exist backend: 16175
2024-02-04 04:08:22,384 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 16212 with slots num: 0
2024-02-04 04:08:22,384 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 16213 with slots num: 0
2024-02-04 04:08:22,384 INFO (tablet scheduler|29) [TabletScheduler.updateWorkingSlots():223] add new backend 16214 with slots num: 0
2024-02-04 04:08:22,917 WARN (heartbeat-mgr-pool-2|140) [HeartbeatMgr$BackendHeartbeatHandler.call():266] backend heartbeat got exception
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:255) ~[libthrift-0.16.0.jar:0.16.0]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
    at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
    at org.apache.thrift.transport.TSocket.open(TSocket.java:250) ~[libthrift-0.16.0.jar:0.16.0]
    ... 13 more
...
2024-02-04 04:08:22,918 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10002, beHost: doris-cluster-doris-be-0.doris-cluster-doris-be-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:22,917 WARN (heartbeat-mgr-pool-6|144) [HeartbeatMgr$BackendHeartbeatHandler.call():266] backend heartbeat got exception
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:255) ~[libthrift-0.16.0.jar:0.16.0]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
    at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
    at org.apache.thrift.transport.TSocket.open(TSocket.java:250) ~[libthrift-0.16.0.jar:0.16.0]
    ... 13 more
2024-02-04 04:08:22,918 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10003, beHost: doris-cluster-doris-be-1.doris-cluster-doris-be-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:22,919 WARN (heartbeat-mgr-pool-3|141) [HeartbeatMgr$BackendHeartbeatHandler.call():266] backend heartbeat got exception
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:255) ~[libthrift-0.16.0.jar:0.16.0]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
    at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
    at org.apache.thrift.transport.TSocket.open(TSocket.java:250) ~[libthrift-0.16.0.jar:0.16.0]
    ... 13 more
2024-02-04 04:08:22,919 WARN (heartbeat-mgr-pool-1|139) [HeartbeatMgr$BackendHeartbeatHandler.call():266] backend heartbeat got exception
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:255) ~[libthrift-0.16.0.jar:0.16.0]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
    at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
    at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
    at org.apache.thrift.transport.TSocket.open(TSocket.java:250) ~[libthrift-0.16.0.jar:0.16.0]
    ... 13 more
2024-02-04 04:08:22,920 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10004, beHost: doris-cluster-doris-be-2.doris-cluster-doris-be-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:22,920 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 16212, beHost: doris-cluster-doris-cn-0.doris-cluster-doris-cn-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:22,920 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 16213, beHost: doris-cluster-doris-cn-1.doris-cluster-doris-cn-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0
2024-02-04 04:08:22,920 WARN (heartbeat mgr|25) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 16214, beHost: doris-cluster-doris-cn-2.doris-cluster-doris-cn-headless.default.svc.cluster.local, bePort: 0, httpPort: 0, brpcPort: 0

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

shanshanying commented 4 months ago

Does not work on GKE. ( cannot access hostpath /root/.kube) Will check if there is any workaround. img_v3_02c9_6c693826-82af-4efd-951b-e4d3b0c8e87g

shanshanying commented 4 months ago

And the err msg here shows that:

fail to find one valid address, exit.

Will checkout the root cause.

shanshanying commented 4 months ago
  1. intall KB 082
  2. install doris addon
  3. install doris cluster (set clusterVersion to doris-2.0.3) But found the FE and BE are not working as expected image image
shanshanying commented 4 months ago

https://github.com/apache/doris/issues/24292

yuqun20230302 commented 4 months ago

priority networks This parameter addresses is the pod network segment of k8s,Please check

shanshanying commented 4 months ago

priority networks This parameter addresses is the pod network segment of k8s,Please check

Hi @yuqun20230302

What is the K8s version you are working witih?

shanshanying commented 4 months ago

priority networks This parameter addresses is the pod network segment of k8s,Please check

It works. Thanks.

image
shanshanying commented 4 months ago