getGitxxx
commented
9 years ago this is log info
22-01-2015 18:14:21 CST hive-demo INFO - Starting job hive-demo at 1421921661351 22-01-2015 18:14:21 CST hive-demo INFO - Building hive job executor. 22-01-2015 18:14:21 CST hive-demo INFO - Not setting up secure proxy info for child process 22-01-2015 18:14:21 CST hive-demo INFO - No classpath specified. Trying to load classes from /app/azkaban/azkaban-executor-2.5.0/executions/149 22-01-2015 18:14:21 CST hive-demo INFO - 1 commands to execute. 22-01-2015 18:14:21 CST hive-demo INFO - Command: java -Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-azkaban -Dhive.aux.jars.path=file:///home/sinova/hive-0.14.0/aux/lib -Xms64M -Xmx256M -cp /app/azkaban/azkaban-executor-2.5.0/lib/azkaban-2.5.0.jar:/app/azkaban/azkaban-executor-2.5.0/plugins/jobtypes/hive/azkaban-jobtype-2.5.jar:/app/azkaban/azkaban-executor-2.5.0/plugins/jobtypes/hive/azkaban-hadoopsecuritymanager-2.5.jar:/home/sinova/hadoop-1.0.1/conf:/home/sinova/hadoop-1.0.1/lib/:/home/sinova/hive-0.14.0/lib/:/home/sinova/hive-0.14.0/conf:/app/azkaban/azkaban-executor-2.5.0/plugins/jobtypes/hive/file:/home/sinova/hive-0.14.0/aux/lib:/home/sinova/hadoop-1.0.1/hadoop-core-1.0.1.jar azkaban.jobtype.HadoopSecureHiveWrapper -hiveconf 'dt=20150122' -f hive-load-part.hql 22-01-2015 18:14:21 CST hive-demo INFO - Environment variables: {JOB_NAME=hive-demo, HIVE_HOME=$HIVE_HOME, JOB_PROP_FILE=/app/azkaban/azkaban-executor-2.5.0/executions/149/hive-demo_props_1032021434949287271_tmp, JOB_OUTPUT_PROP_FILE=/app/azkaban/azkaban-executor-2.5.0/executions/149/hive-demo_output_4340661934105694333_tmp, HIVE_AUX_JARS_PATH=file:///home/sinova/hive-0.14.0/aux/lib} 22-01-2015 18:14:21 CST hive-demo INFO - Working directory: /app/azkaban/azkaban-executor-2.5.0/executions/149 22-01-2015 18:14:21 CST hive-demo INFO - 2015/01/22 18:14:21.899 +0800 INFO [root] [Azkaban] Not proxying. 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] HiveConf = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@249ef1e, file:/home/sinova/hive-0.14.0/conf/hive-site.xml 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] According to the conf, we're talking to the Hive hosted at: jdbc:mysql://10.20.11.17:3306/hive13?createDatabaseIfNotExist=true 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] No files in to expand in aux jar path. Returning original parameter 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] Hive aux jars variable not expanded 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.289 +0800 INFO [root] [Azkaban] Got auxJars = file:///home/sinova/hive-0.14.0/aux/lib 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.454 +0800 INFO [metastore] [Azkaban] Trying to connect to metastore with URI thrift://10.40.33.11:9083 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.504 +0800 INFO [metastore] [Azkaban] Connected to metastore. 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.824 +0800 WARN [NativeCodeLoader] [Azkaban] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.828 +0800 INFO [SessionState] [Azkaban] Created local directory: /tmp/38c7abd4-70c2-41ba-94c5-541b95722644_resources 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.881 +0800 INFO [SessionState] [Azkaban] Created HDFS directory: /tmp/hive-azkaban/sinova/38c7abd4-70c2-41ba-94c5-541b95722644 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.885 +0800 INFO [SessionState] [Azkaban] Created local directory: /tmp/sinova/38c7abd4-70c2-41ba-94c5-541b95722644 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.893 +0800 INFO [SessionState] [Azkaban] Created HDFS directory: /tmp/hive-azkaban/sinova/38c7abd4-70c2-41ba-94c5-541b95722644/_tmp_space.db 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.895 +0800 INFO [SessionState] [Azkaban] No Tez session required at this point. hive.execution.engine=mr. 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.896 +0800 INFO [root] [Azkaban] SessionState = org.apache.hadoop.hive.cli.CliSessionState@4777eb45 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.897 +0800 INFO [root] [Azkaban] Executing query: hive-load-part.hql 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.931 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.931 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.932 +0800 INFO [Driver] [Azkaban] Concurrency mode is disabled, not creating a lock manager 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.932 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.975 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.984 +0800 INFO [ParseDriver] [Azkaban] Parsing command: LOAD DATA LOCAL INPATH '/app/azkaban/web2' OVERWRITE INTO TABLE words PARTITION (day_id='${hiveconf:dt}') 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.376 +0800 INFO [ParseDriver] [Azkaban] Parse Completed 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.377 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.381 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.631 +0800 INFO [Driver] [Azkaban] Semantic Analysis Completed 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.631 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [Driver] [Azkaban] Returning Hive schema: Schema(fieldSchemas:null, properties:null) 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [Driver] [Azkaban] Starting command: LOAD DATA LOCAL INPATH '/app/azkaban/web2' OVERWRITE INTO TABLE words PARTITION (day_id='${hiveconf:dt}') 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.644 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.644 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.644 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.645 +0800 INFO [Driver] [Azkaban] Starting task [Stage-0:MOVE] in serial mode 22-01-2015 18:14:23 CST hive-demo ERROR - Loading data to table default.words partition (day_id=${hiveconf:dt}) 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.646 +0800 INFO [Task] [Azkaban] Loading data to table default.words partition (day_id=${hiveconf:dt}) from file:/app/azkaban/web2 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.687 +0800 INFO [MoveTask] [Azkaban] Partition is: {day_id=${hiveconf:dt}} 22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.954 +0800 INFO [Hive] [Azkaban] Replacing src:file:/app/azkaban/web2;dest: hdfs://bj-ecs-jm429-I-e2-1-bl460c-9-null-9:9000/user/sinova/hive/warehouse/words/day_id=$%7Bhiveconf%3Adt}/web2;Status:true 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.255 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.256 +0800 INFO [Driver] [Azkaban] Starting task [Stage-1:STATS] in serial mode 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.256 +0800 INFO [StatsTask] [Azkaban] Executing stats task 22-01-2015 18:14:24 CST hive-demo ERROR - Partition default.words{day_id=${hiveconf:dt}} stats: [numFiles=1, numRows=0, totalSize=2728995, rawDataSize=0] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.368 +0800 INFO [Task] [Azkaban] Partition default.words{day_id=${hiveconf:dt}} stats: [numFiles=1, numRows=0, totalSize=2728995, rawDataSize=0] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.508 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.508 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo ERROR - OK 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.509 +0800 INFO [Driver] [Azkaban] OK 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.510 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.510 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.510 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo ERROR - Time taken: 1.583 seconds 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.512 +0800 INFO [CliDriver] [Azkaban] Time taken: 1.583 seconds 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.512 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.512 +0800 INFO [PerfLogger] [Azkaban] 22-01-2015 18:14:24 CST hive-demo INFO - Process completed successfully in 3 seconds. 22-01-2015 18:14:24 CST hive-demo INFO - Finishing job hive-demo at 1421921664644 with status SUCCEEDED
hi all I use hive plugin on azkaban v2.5,I need some dynamic params to hiveQL files without azkaban,I set hiveconf like this bin/hive -f “path/to/hiveql/test.q” -hiveconf tabname=table1 in test.q,i use params like this select * from ${hiveconf:tabname} this hive job works well.
so .my requestion is how to use hive like this in Azkaban?
Any relay will greatly appreciated!