search

Isilon / isilon_hadoop_tools

Tools for Using Hadoop with OneFS
https://pypi.org/project/isilon-hadoop-tools/
MIT License
14 stars 21 forks source link

Issue with the Script #95

Open arjunnandy opened 3 years ago

arjunnandy commented 3 years ago

The Isilon_create_directories script has an issue where it did not create the warehouse/tablespace/external. Mentioned below is a dump of the debug after executing the script

[2021-04-23 09:44:18,076] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' [2021-04-23 09:44:18,076] isilon_hadoop_tools.directories [INFO] chmod '775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' [2021-04-23 09:44:18,076] isilon_hadoop_tools.directories [INFO] chown 'hdfs:supergroup' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chmod '775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chown 'hdfs:supergroup' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external/hive' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chmod '1775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external/hive' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chown 'hive:hive' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external/hive' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed/hive' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chmod '1775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed/hive' [2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chown 'hive:hive' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed/hive' [2021-04-23 09:47:27,628] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' "message" : "A component of path 'data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' does not exist." [2021-04-23 09:47:28,496] root [ERROR] A component of path 'data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' does not exist.

After running the script we went to the Isilon and verified that the tablespace directory was missing under warehouse. ncwdh5icl11-1# cd /ifs/data/UAT_SAAP/NCW/HDFS/warehouse/ ncwdh5icl11-1# ls ncwdh5icl11-1# pwd

The customer had to create the tablespace directory manually. I too have noticed the same behavior. Could you correct the script so that the error does not come up.

shirleylxie commented 3 years ago

This issue should be fixed with v4.1.3. Please have a try at the newest version.

Here is the log of v4.1.3.

[INFO] chown 'hdfs:supergroup' '/ifs/data/warehouse/tablespace'
[INFO] mkdir '/ifs/data/warehouse/tablespace/external'
[INFO] chmod '775' '/ifs/data/warehouse/tablespace/external'
[INFO] chown 'hdfs:supergroup' '/ifs/data/warehouse/tablespace/external'
[INFO] mkdir '/ifs/data/warehouse/tablespace/managed'
[INFO] chmod '775' '/ifs/data/warehouse/tablespace/managed'
[INFO] chown 'hdfs:supergroup' '/ifs/data/warehouse/tablespace/managed'
[INFO] mkdir '/ifs/data/warehouse/tablespace/external/hive'
[INFO] chmod '1775' '/ifs/data/warehouse/tablespace/external/hive'
[INFO] chown 'hive:hive' '/ifs/data/warehouse/tablespace/external/hive'
[INFO] mkdir '/ifs/data/warehouse/tablespace/managed/hive'
[INFO] chmod '1775' '/ifs/data/warehouse/tablespace/managed/hive'
[INFO] chown 'hive:hive' '/ifs/data/warehouse/tablespace/managed/hive'
tucked commented 3 years ago

Duplicate of #94