search

Isilon / isilon_hadoop_tools

Tools for Using Hadoop with OneFS
https://pypi.org/project/isilon-hadoop-tools/
MIT License
14 stars 21 forks source link

User and directory scripts overlap on user folders #36

Closed ro-ket closed 7 years ago

ro-ket commented 7 years ago

The user creation script creates a folders for each user as $HDFSROOT/user/$user during the isi auth users create command.

The directory creation script attempts to create some of those folders as well. (It has an incomplete list of users.) When it tries to create them it issues errors, surprised that the folders exist already.

Here is partial output of the directory creation script for --dist=cdh after running the user creation script:

anisiloncluster-1# ./isilon_create_directories.sh --dist cdh --zone System
Info: HDFS root dir is /ifs/hdfs-root
...
DEBUG: specs dirname /user; perm 755; owner hdfs; group supergroup
ERROR:  Directory /ifs/hdfs-root/user exists but no --fixperm not specified
DEBUG: specs dirname /user/history; perm 777; owner mapred; group hadoop
DEBUG: specs dirname /user/hive; perm 775; owner hive; group hive
ERROR:  Directory /ifs/hdfs-root/user/hive exists but no --fixperm not specified
DEBUG: specs dirname /user/hive/warehouse; perm 1777; owner hive; group hive
DEBUG: specs dirname /user/hue; perm 755; owner hue; group hue
ERROR:  Directory /ifs/hdfs-root/user/hue exists but no --fixperm not specified
DEBUG: specs dirname /user/hue/.cloudera_manager_hive_metastore_canary; perm 777; owner hue; group hue
DEBUG: specs dirname /user/impala; perm 775; owner impala; group impala
ERROR:  Directory /ifs/hdfs-root/user/impala exists but no --fixperm not specified
DEBUG: specs dirname /user/oozie; perm 775; owner oozie; group oozie
ERROR:  Directory /ifs/hdfs-root/user/oozie exists but no --fixperm not specified
DEBUG: specs dirname /user/flume; perm 775; owner flume; group flume
ERROR:  Directory /ifs/hdfs-root/user/flume exists but no --fixperm not specified
DEBUG: specs dirname /user/spark; perm 751; owner spark; group spark
ERROR:  Directory /ifs/hdfs-root/user/spark exists but no --fixperm not specified
DEBUG: specs dirname /user/spark/applicationHistory; perm 1777; owner spark; group spark
DEBUG: specs dirname /user/sqoop2; perm 775; owner sqoop2; group sqoop
ERROR:  Directory /ifs/hdfs-root/user/sqoop2 exists but no --fixperm not specified
DEBUG: specs dirname /solr; perm 775; owner solr; group solr
ERROR:  Directory /ifs/hdfs-root/solr exists but no --fixperm not specified
ERRORS FOUND:
...
ERROR:  Directory /ifs/hdfs-root/user exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/hive exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/hue exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/impala exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/oozie exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/flume exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/spark exists but no --fixperm not specified
ERROR:  Directory /ifs/hdfs-root/user/sqoop2 exists but no --fixperm not specified
...
FATAL:  ERRORS FOUND making Hadoop admin directory structure -- please fix before continuing

Note that these folders were not created by the user creation script: