search

Isilon / isilon_hadoop_tools

Tools for Using Hadoop with OneFS
https://pypi.org/project/isilon-hadoop-tools/
MIT License
14 stars 21 forks source link

BI doesn't seem to work on a fresh HP cluster #19

Closed dmtucker closed 7 years ago

dmtucker commented 7 years ago 
./isilon_create_users.sh --dist bi
Info: Hadoop distribution:  bi
Info: HDFS root:  /ifs/hdfs
Info: passwd file: System.passwd
Info: group file: System.group
The Name livy is not resolvable
The Name HTTP is not resolvable
The Name ambari-server is not resolvable
Failed to create persona 'USER:kafka'
ERRORS FOUND:
*  ERROR:  Could not create proxyuser livy in zone System
*  ERROR:  Could not create proxyuser HTTP in zone System
*  ERROR:  Could not create proxyuser ambari-server in zone System
*  ERROR:  Could not add user kafka to knox group in zone System
FATAL:  ERRORS FOUND making Hadoop users in zone System -- please fix before continuing

Looks like 3abfd6d4e4a0027b6dc5c1b94ae9142cd6abadba, 6cbf7f15d4323aa390b86f8a4832b8654684ac24, and claudiofahey/isilon-hadoop-tools@b5d12d409348024aaf4053f2931afdd305b40175 should have added HTTP, livy/ambari-server, and kafka to REQUIRED_USERS, respectively. In fact, claudiofahey/isilon-hadoop-tools@9f3d7e96e2467f2489a16d8e6d5d36785411afec may have been an attempt to fix the latter.

dmtucker commented 7 years ago

Another thing to note is that kafka gets put into the knox group in the group file even though that failed on OneFS:

# use this file to add to the group file of your clients
hdfs:x:1000:
mapred:x:1001:
hbase:x:1002:
uiuser:x:1004:
dsmadmin:x:1005:
bigsheets:x:1006:
ambari-qa:x:1007:
rrdcached:x:1008:
hive:x:1009:
yarn:x:1010:
bigsql:x:1012:
tauser:x:1013:
bigr:x:1014:
flume:x:1015:
nagios:x:1016:
solr:x:1017:
spark:x:1018:
sqoop:x:1019:
zookeeper:x:1020:
oozie:x:1021:
bighome:x:1022:
ams:x:1023:
anonymous:x:1024:
hadoop:x:1025:,hdfs,mapred,hbase,knox,uiuser,dsmadmin,bigsheets,ambari-qa,rrdcached,hive,yarn,hcat,bigsql,tauser,bigr,flume,nagios,solr,spark,sqoop,zookeeper,oozie,bighome,ams
hcat:x:1011:,hive
knox:x:1003:,kafka

This could be intended or not... we did try after all. Just thought I'd point it out.

bonibruno commented 7 years ago

There were some missing users. Try this version out and let me know...

dmtucker commented 7 years ago

4df24d13973c44de6211a5a1aa3ffda140a06187 seems to have made things worse:

Info: Hadoop distribution:  bi
Info: HDFS root:  /ifs/hdfs
Info: passwd file: System.passwd
Info: group file: System.group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to find group for 'GROUP:hadoop': No such group
Failed to create persona 'USER:ambari-server'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
Failed to create persona 'GROUP:hadoop'
The Name ambari-server is not resolvable
Failed to create persona 'USER:kafka'
ERRORS FOUND:
*  ERROR:  Group knox already exists at gid 1003 in zone System
*  ERROR:  GID 1004 already in use by group uiuser in zone System
*  ERROR:  GID 1005 already in use by group dsmadmin in zone System
*  ERROR:  GID 1006 already in use by group bigsheets in zone System
*  ERROR:  User knox already exists at uid 1003 in zone System
*  ERROR:  UID 1004 already in use by user uiuser in zone System
*  ERROR:  UID 1005 already in use by user dsmadmin in zone System
*  ERROR:  Could not locate entry hadoop in group file stub System.group
*  ERROR:  Could not add user hdfs to hadoop group in zone System
*  ERROR:  Could not add user mapred to hadoop group in zone System
*  ERROR:  Could not add user hbase to hadoop group in zone System
*  ERROR:  Could not add user knox to hadoop group in zone System
*  ERROR:  Could not add user uiuser to hadoop group in zone System
*  ERROR:  Could not add user dsmadmin to hadoop group in zone System
*  ERROR:  Could not add user bigsheets to hadoop group in zone System
*  ERROR:  Could not add user ambari-qa to hadoop group in zone System
*  ERROR:  Could not add user rrdcached to hadoop group in zone System
*  ERROR:  Could not add user hive to hadoop group in zone System
*  ERROR:  Could not add user yarn to hadoop group in zone System
*  ERROR:  Could not add user hcat to hadoop group in zone System
*  ERROR:  Could not add user bigsql to hadoop group in zone System
*  ERROR:  Could not add user tauser to hadoop group in zone System
*  ERROR:  Could not add user bigr to hadoop group in zone System
*  ERROR:  Could not add user flume to hadoop group in zone System
*  ERROR:  Could not add user nagios to hadoop group in zone System
*  ERROR:  Could not add user solr to hadoop group in zone System
*  ERROR:  Could not add user spark to hadoop group in zone System
*  ERROR:  Could not add user sqoop to hadoop group in zone System
*  ERROR:  Could not add user zookeeper to hadoop group in zone System
*  ERROR:  Could not add user oozie to hadoop group in zone System
*  ERROR:  Could not add user bighome to hadoop group in zone System
*  ERROR:  Could not add user ams to hadoop group in zone System
*  ERROR:  Could not add user livy to hadoop group in zone System
*  ERROR:  Could not add user logsearch to hadoop group in zone System
*  ERROR:  Could not add user infra-solr to hadoop group in zone System
*  ERROR:  Could not add user activity_analyzer to hadoop group in zone System
*  ERROR:  Could not add user activity_explorer to hadoop group in zone System
*  ERROR:  Could not add user HTTP to hadoop group in zone System
*  ERROR:  Could not add user knox to hadoop group in zone System
*  ERROR:  Could not add user ambari-server to hadoop group in zone System
*  ERROR:  Could not create proxyuser yarn in zone System
*  ERROR:  Could not create proxyuser livy in zone System
*  ERROR:  Could not create proxyuser hcat in zone System
*  ERROR:  Could not create proxyuser hbase in zone System
*  ERROR:  Could not create proxyuser flume in zone System
*  ERROR:  Could not create proxyuser hive in zone System
*  ERROR:  Could not create proxyuser oozie in zone System
*  ERROR:  Could not create proxyuser root in zone System
*  ERROR:  Could not create proxyuser ambari-server in zone System
*  ERROR:  Could not add user kafka to knox group in zone System
FATAL:  ERRORS FOUND making Hadoop users in zone System -- please fix before continuing