正常安装hdfs时出现以下报错，namenode无法正常安装

taoqibao0513 commented 1 month ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

所用mysql版本为5.7.26版本，所用datasophon1.2.1版本，所用DDP包为1.2.1版本，安装环境为默认官网环境，但无法正常安装hdfs。开始安装hdfs时出现zkfc报错 [ERROR] 2024-05-20 10:38:53 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 10:38:53 TaskLogLogger-HDFS-ZKFC:[182] - . control_hadoop.sh: line 61: kill: (14800) - No such process zkfc is not running

[ERROR] 2024-05-20 10:38:53 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 10:38:53 TaskLogLogger-HDFS-ZKFC:[175] - execute shell command : [sudo, -u, hdfs, bash, control_hadoop.sh, start, zkfc] [INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[182] - . execute ./bin/hdfs --daemon start zkfc zkfc start success End start zkfc.

[INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[154] - script execute success [ERROR] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[71] - check start result at times 1 [INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[175] - execute shell command : [sudo, -u, hdfs, bash, control_hadoop.sh, status, zkfc] [INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[182] - . zkfc is running End status zkfc.

[ERROR] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[154] - script execute success [INFO] 2024-05-20 10:38:55 TaskLogLogger-HDFS-ZKFC:[74] - start success in hadoop-3.3.3 [INFO] 2024-05-20 10:59:07 TaskLogLogger-HDFS-ZKFC:[175] - execute shell command : [sudo, -u, hdfs, bash, control_hadoop.sh, status, zkfc] [INFO] 2024-05-20 10:59:07 TaskLogLogger-HDFS-ZKFC:[182] - . zkfc is running End status zkfc.

[ERROR] 2024-05-20 10:59:07 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 10:59:07 TaskLogLogger-HDFS-ZKFC:[154] - script execute success [INFO] 2024-05-20 10:59:07 TaskLogLogger-HDFS-ZKFC:[175] - execute shell command : [sudo, -u, hdfs, bash, control_hadoop.sh, stop, zkfc] [INFO] 2024-05-20 10:59:08 TaskLogLogger-HDFS-ZKFC:[182] - . zkfc stop success End stop zkfc.

[INFO] 2024-05-20 10:59:08 TaskLogLogger-HDFS-ZKFC:[154] - script execute success [ERROR] 2024-05-20 10:59:08 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 10:59:08 TaskLogLogger-HDFS-ZKFC:[104] - check stop result at times 1 [INFO] 2024-05-20 10:59:08 TaskLogLogger-HDFS-ZKFC:[175] - execute shell command : [sudo, -u, hdfs, bash, control_hadoop.sh, status, zkfc] [INFO] 2024-05-20 10:59:09 TaskLogLogger-HDFS-ZKFC:[182] - . zkfc pid file is not exists

[INFO] 2024-05-20 10:59:09 TaskLogLogger-HDFS-ZKFC:[107] - stop success in hadoop-3.3.3 [ERROR] 2024-05-20 10:59:09 TaskLogLogger-HDFS-ZKFC:[197] - [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[82] - Start to configure service role ZKFC [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[263] - size is :1 [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[266] - config set value to /data/dfs/nn [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[263] - size is :1 [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[266] - config set value to /data/dfs/dn [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[263] - size is :4 [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[266] - config set value to RULE:2:$1/$2@$0s/./hdfs/ RULE:[2:$1/$2@$0]([rn]m\/.@HADOOP.COM)s/./yarn/ RULE:[2:$1/$2@$0](jhs\/.@HADOOP.COM)s/./mapred/ DEFAULT [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 11:24:06 TaskLogLogger-HDFS-ZKFC:[41] - start to execute hdfs zkfc -formatZK [INFO] 2024-05-20 11:29:06 TaskLogLogger-HDFS-ZKFC:[53] - zkfc format failed [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[82] - Start to configure service role ZKFC [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[263] - size is :1 [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[266] - config set value to /data/dfs/nn [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[263] - size is :1 [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[266] - config set value to /data/dfs/dn [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[117] - Convert boolean and integer to string [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[263] - size is :4 [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[266] - config set value to RULE:[2:$1/$2@$0]([ndj]n\/.@HADOOP.COM)s/./hdfs/ RULE:[2:$1/$2@$0]([rn]m\/.@HADOOP.COM)s/./yarn/ RULE:[2:$1/$2@$0](jhs\/.@HADOOP.COM)s/.*/mapred/ DEFAULT [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[181] - configure success [INFO] 2024-05-20 13:18:07 TaskLogLogger-HDFS-ZKFC:[41] - start to execute hdfs zkfc -formatZK [INFO] 2024-05-20 13:18:08 TaskLogLogger-HDFS-ZKFC:[182] - [31m 欢迎来到Redhat Linux 操作系统[0m [1;31m 请合理使用云主机分配内存，所有应用目录建议使用/data目录;[0m [1;31m 系统后期维护，建议不要使用lvm方式扩容根(/)目录. [0m [1;31m 系统未分配swap，请自行添加swap分区，百度即可，swap分区有利于节约内存开销；[0m zkfc is running as process 17853. Stop it first and ensure /opt/datasophon/hadoop-3.3.3/pid/hadoop-hdfs-zkfc.pid file is empty before retry. Last login: Mon May 20 13:18:07 CST 2024

[INFO] 2024-05-20 13:18:08 TaskLogLogger-HDFS-ZKFC:[53] - zkfc format failed [ERROR] 2024-05-20 13:18:08 TaskLogLogger-HDFS-ZKFC:[197] -

还没有跳到其他地方已经停止了安装工作。 can not find log file 只能成功安装 JournalNode，并且在查看zookeeper时，出现经常报错，得等待很长时间才能安装成功。

What you expected to happen

安装zkfc失败后，没有namenode，无法安装yarn，导致后续一大堆组件无法安装。希望可以通过官方安装，正确引导下安装成功hdfs及后续的组件。

How to reproduce

按照官方流程，正常搭建服务后，出现此类情况。

Anything else

每次在搭建时都会出现。

Version

v1.1.2

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

datasophon commented 1 month ago

这看起来是你的操作系统不支持安装zkfc

taoqibao0513 commented 1 month ago

这看起来是你的操作系统不支持安装zkfc 你好，我这边运行的linux版本为：Linux version 3.10.0-1160.el7.x86_64 (mockbuild@x86-vm-26.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Tue Aug 18 14:50:17 EDT 2020，操作系统的版本为Red Hat Enterprise Linux Server release 7.9 (Maipo)，请问有什么改进的建议么

datasophon commented 1 month ago

看起来是启动脚本在你的系统中无法执行

datavane / datasophon