search

XiaoMi / minos

Minos is beyond a hadoop deployment system.
Apache License 2.0
522 stars 200 forks source link

hdfs-dptst-example.cfg配置文件相关参数问题咨询 #27

Open xiaoguanyu opened 10 years ago

xiaoguanyu commented 10 years ago

文件目录：monos/config/conf/hdfs/hdfs-dptst-example.cfg [journalnode] base_port=12100 host.0=10.38.11.59 host.1=10.38.11.134 host.2=10.38.11.135 [namenode] base_port=12200 host.0=10.38.11.59 host.1=10.38.11.134 [zkfc] base_port=12300 [datanode] base_port=12400 host.0=10.38.11.134 host.1=10.38.11.135 请问这些参数中base_port的设置是同hadoop的配置文件中的journalnode、namenode等的rpc端口号相同，还是自己随意定义的

YxAc commented 10 years ago

会根据base_port的设置生成hadoop的配置文件，rpc端口对应着base_port

2014-05-15 9:07 GMT+08:00 xiaoguanyu notifications@github.com:

文件目录：monos/config/conf/hdfs/hdfs-dptst-example.cfg [journalnode] base_port=12100 host.0=10.38.11.59 host.1=10.38.11.134 host.2=10.38.11.135 [namenode] base_port=12200 host.0=10.38.11.59 host.1=10.38.11.134 [zkfc] base_port=12300 [datanode] base_port=12400 host.0=10.38.11.134 host.1=10.38.11.135 请问这些参数中base_port的设置是同hadoop的配置文件中的journalnode、namenode等的rpc端口号相同，还是自己随意定义的

— Reply to this email directly or view it on GitHubhttps://github.com/XiaoMi/minos/issues/27 .

xiaoguanyu commented 10 years ago

但是我已经在集群中部署了hadoop2，如果根据base_port的设置生成的hadoop配置文件是hadoop2/etc/hadoop/hdfs-site.xml文件吗？

xiaoguanyu_java@163.com

发件人： 勇幸 发送时间： 2014-05-15 09:35 收件人： XiaoMi/minos 抄送： xiaoguanyu 主题： Re: [minos] hdfs-dptst-example.cfg配置文件相关参数问题咨询 (#27) 会根据base_port的设置生成hadoop的配置文件，rpc端口对应着base_port

2014-05-15 9:07 GMT+08:00 xiaoguanyu notifications@github.com:

文件目录：monos/config/conf/hdfs/hdfs-dptst-example.cfg [journalnode] base_port=12100 host.0=10.38.11.59 host.1=10.38.11.134 host.2=10.38.11.135 [namenode] base_port=12200 host.0=10.38.11.59 host.1=10.38.11.134 [zkfc] base_port=12300 [datanode] base_port=12400 host.0=10.38.11.134 host.1=10.38.11.135 请问这些参数中base_port的设置是同hadoop的配置文件中的journalnode、namenode等的rpc端口号相同，还是自己随意定义的

— Reply to this email directly or view it on GitHubhttps://github.com/XiaoMi/minos/issues/27 .

— Reply to this email directly or view it on GitHub.

xiaoguanyu commented 10 years ago

minos的client是什么原理只要执行./deploy install zookeeper dptst 和 ./deploy install hdfs dptst-example 即使集群中部署了zk和hdfs，也会自动安装一个zk和hdfs吗？

xiaoguanyu_java@163.com

发件人： 勇幸 发送时间： 2014-05-15 09:35 收件人： XiaoMi/minos 抄送： xiaoguanyu 主题： Re: [minos] hdfs-dptst-example.cfg配置文件相关参数问题咨询 (#27) 会根据base_port的设置生成hadoop的配置文件，rpc端口对应着base_port

2014-05-15 9:07 GMT+08:00 xiaoguanyu notifications@github.com:

文件目录：monos/config/conf/hdfs/hdfs-dptst-example.cfg [journalnode] base_port=12100 host.0=10.38.11.59 host.1=10.38.11.134 host.2=10.38.11.135 [namenode] base_port=12200 host.0=10.38.11.59 host.1=10.38.11.134 [zkfc] base_port=12300 [datanode] base_port=12400 host.0=10.38.11.134 host.1=10.38.11.135 请问这些参数中base_port的设置是同hadoop的配置文件中的journalnode、namenode等的rpc端口号相同，还是自己随意定义的

— Reply to this email directly or view it on GitHubhttps://github.com/XiaoMi/minos/issues/27 .

— Reply to this email directly or view it on GitHub.

YxAc commented 10 years ago

你的hadoop2是minos部署的？使用minos生成的hadoop配置文件会放在集群机器上job的run路径下,例如/home/work/app/hdfs/dptst-example/journalnode/hdfs-site.xml client的install命令只是将hadoop包上传到Tank服务器上；在bootstrap的时候，各个production machine会从tank上拉取包；集群中是否部署zk和hdfs与install没有直接关系，只是必须先install将包上传，然后才可以bootstrap

xiaoguanyu commented 10 years ago

我的hadoop2是自己部署的，我理解错了，我以为是执行install就会自动安装hadoop,我将client/supervisor_client.py中的 self.proxy self.service self.cluster
self.job self.instance_id 都打印出来了，总是报连接错误 [root@namenode client]# ./deploy bootstrap hdfs dptst-example 2014-05-15 10:38:59 You should set a bootstrap password, it will be requried when you do cleanup Set a password manually? (y/n) y Please input your password: 2014-05-15 10:39:03 Your password is: 123456, you should store this in a safe place, because this is the verification code used to do cleanup <ServerProxy for root:zxcvbnm@10.38.11.59:9001/RPC2> hdfs dptst-example journalnode -1 <ServerProxy for root:zxcvbnm@10.38.11.59:9001/RPC2> hdfs dptst-example namenode -1 <ServerProxy for root:zxcvbnm@10.38.11.59:9001/RPC2> hdfs dptst-example namenode -1 <ServerProxy for root:zxcvbnm@10.38.11.59:9001/RPC2> hdfs dptst-example namenode -1 <ServerProxy for root:zxcvbnm@10.38.11.59:9001/RPC2> hdfs dptst-example namenode -1 <ServerProxy for root:zxcvbnm@10.38.11.134:9001/RPC2> hdfs dptst-example datanode -1 Traceback (most recent call last): File "/usr/local/test/minos/client/deploy.py", line 284, in main() File "/usr/local/test/minos/client/deploy.py", line 281, in main return args.handler(args) File "/usr/local/test/minos/client/deploy.py", line 229, in process_command_bootstrap return deploy_tool.bootstrap(args) File "/usr/local/test/minos/client/deploy_hdfs.py", line 238, in bootstrap bootstrap_job(args, hosts[host_id].ip, job_name, host_id, instance_id, first, cleanup_token) File "/usr/local/test/minos/client/deploy_hdfs.py", line 201, in bootstrap_job args.hdfs_config.parse_generated_config_files(args, job_name, host_id, instance_id) File "/usr/local/test/minos/client/service_config.py", line 665, in parse_generated_config_files args, self.cluster, self.jobs, current_job, host_id, instance_id)) File "/usr/local/test/minos/client/service_config.py", line 652, in parse_generated_files file_dict[key] = ServiceConfig.parse_item(args, cluster, jobs, current_job, host_id, instance_id, value) File "/usr/local/test/minos/client/service_config.py", line 596, in parse_item new_item.append(callback(args, cluster, jobs, current_job, host_id, instance_id, reg_expr[iter])) File "/usr/local/test/minos/client/service_config.py", line 255, in get_section_attribute return get_specific_dir(host.ip, args.service, cluster.name, section, section_instance_id, attribute) File "/usr/local/test/minos/client/service_config.py", line 185, in get_specific_dir return ",".join(supervisor_client.get_available_data_dirs()) File "/usr/local/test/minos/client/supervisor_client.py", line 30, in get_available_data_dirs self.cluster, self.job) File "/usr/local/lib/python2.7/xmlrpclib.py", line 1224, in call return self.send(self.name, args) File "/usr/local/lib/python2.7/xmlrpclib.py", line 1578, in request verbose=self.verbose File "/usr/local/lib/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/local/lib/python2.7/xmlrpclib.py", line 1292, in single_request self.send_content(h, request_body) File "/usr/local/lib/python2.7/xmlrpclib.py", line 1439, in send_content connection.endheaders(request_body) File "/usr/local/lib/python2.7/httplib.py", line 969, in endheaders self._send_output(message_body) File "/usr/local/lib/python2.7/httplib.py", line 829, in _send_output self.send(msg) File "/usr/local/lib/python2.7/httplib.py", line 791, in send self.connect() File "/usr/local/lib/python2.7/httplib.py", line 772, in connect self.timeout, self.source_address) File "/usr/local/lib/python2.7/socket.py", line 571, in create_connection raise err socket.error: [Errno 111] Connection refused

xiaoguanyu_java@163.com

发件人： 勇幸 发送时间： 2014-05-15 10:15 收件人： XiaoMi/minos 抄送： xiaoguanyu 主题： Re: [minos] hdfs-dptst-example.cfg配置文件相关参数问题咨询 (#27) 你的hadoop2是minos部署的？使用minos生成的hadoop配置文件会放在集群机器上job的run路径下,例如/home/work/app/hdfs/dptst-example/journalnode/hdfs-site.xml client的install命令只是将hadoop包上传到Tank服务器上；在bootstrap的时候，各个production machine会从tank上拉取包；集群中是否部署zk和hdfs与install没有直接关系，只是必须先install将包上传，然后才可以bootstrap — Reply to this email directly or view it on GitHub.

YxAc commented 10 years ago

连接错误，请问你10.38.11.59:9001能访问不？部署前需要先部署好Tank，并且需要在所有的production machines上部署supervisord

xiaoguanyu commented 10 years ago

您好，请问是不是所有部署supervisord的production machines上也都要部署tank

xiaoguanyu_java@163.com

发件人： 勇幸 发送时间： 2014-05-15 11:41 收件人： XiaoMi/minos 抄送： xiaoguanyu 主题： Re: [minos] hdfs-dptst-example.cfg配置文件相关参数问题咨询 (#27) 连接错误，请问你10.38.11.59:9001能访问不？部署前需要先部署好Tank，并且需要在所有的production machines上部署supervisord — Reply to this email directly or view it on GitHub.

wuzesheng commented 10 years ago

tank只需要一个就可以，supervisord需要在每以机器上布署，详细参考readme.md里的架构图

xiaoguanyu commented 10 years ago

我只在10.38.11.59的机器上部署了tank和supervisord,我在10.38.11.8只部署了supervisord，两台机器共享59上的tank，10.38.11.8的supervisor的访问状态如图2所示，显示的失败的两条信息是什么意思？ 10.38.11.59:9001 图1：

图2：10.38.11.8：9001

xiaoguanyu_java@163.com

发件人： Zesheng Wu 发送时间： 2014-05-15 15:32 收件人： XiaoMi/minos 抄送： xiaoguanyu 主题： Re: [minos] hdfs-dptst-example.cfg配置文件相关参数问题咨询 (#27) tank只需要一个就可以，supervisord需要在每以机器上布署，详细参考readme.md里的架构图 — Reply to this email directly or view it on GitHub.

wuzesheng commented 10 years ago

看不到图

xiaoguanyu commented 10 years ago

我只在10.38.11.59的机器上部署了tank和supervisord,我在10.38.11.8只部署了supervisord，两台机器共享59上的tank，10.38.11.8的supervisor的访问状态如图2所示，显示的失败的两条信息是什么意思？请查看附件中图，谢谢 59tu是10.38.11.59supervisor 状态图 8tu是10.38.11.8supervisor 状态图

xiaoguanyu_java@163.com

发件人： Zesheng Wu 发送时间： 2014-05-15 15:47 收件人： XiaoMi/minos 抄送： xiaoguanyu 主题： Re: [minos] hdfs-dptst-example.cfg配置文件相关参数问题咨询 (#27) 看不到图 — Reply to this email directly or view it on GitHub.