Recently I am doing some research in hadoop tools,and I tried white elephant. Everything is ok but uploading log files.
I tested it in my own computer with hadoop1.0.4. And my hadoop is in Pseudo-distributed.
My cfg.pm is below:
"hadoop_home" => "/home/libenchao/hadoop/hadoop-1.0.4"
"hadoop_logs" => "/home/libenchao/hadoop/hadoop-1.0.4/logs"
"days" => 5
"queues" => ['default']
"grid" => 'mygrid'
"destination" => "hdfs://localhost:9000/user/libenchao/history"
When I run: ./statsupload.pl --config cfg.pm
I get:
Checking the last 5 days in HDFS for existing data
/home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1130
/home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1129
/home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1128
/home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1127
/home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1126
/home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1125
Found 0 existing files in HDFS
Searching /home/libenchao/hadoop/hadoop-1.0.4/logs/history for logs
Found no new logs to upload
But this is the first time I upload files,and in my logs folder it does exist log files. I am confused if I am missing something.
I attach my log file folder below:
libenchao@libenchao-Aspire-4741:~/hadoop/hadoop-1.0.4/logs/history$ ls done/version-1/localhost1385716167800/2013/11/29/000000/
job_201311291709_0001_1385739836098_libenchao_word+count
job_201311291709_0001_conf.xml
Recently I am doing some research in hadoop tools,and I tried white elephant. Everything is ok but uploading log files. I tested it in my own computer with hadoop1.0.4. And my hadoop is in Pseudo-distributed. My cfg.pm is below: "hadoop_home" => "/home/libenchao/hadoop/hadoop-1.0.4" "hadoop_logs" => "/home/libenchao/hadoop/hadoop-1.0.4/logs" "days" => 5 "queues" => ['default'] "grid" => 'mygrid' "destination" => "hdfs://localhost:9000/user/libenchao/history"
When I run: ./statsupload.pl --config cfg.pm I get: Checking the last 5 days in HDFS for existing data /home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1130 /home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1129 /home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1128 /home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1127 /home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1126 /home/libenchao/hadoop/hadoop-1.0.4/bin/hadoop dfs -lsr hdfs://localhost:9000/user/libenchao/history/mygrid/daily/default/2013/1125 Found 0 existing files in HDFS Searching /home/libenchao/hadoop/hadoop-1.0.4/logs/history for logs Found no new logs to upload
But this is the first time I upload files,and in my logs folder it does exist log files. I am confused if I am missing something. I attach my log file folder below: libenchao@libenchao-Aspire-4741:~/hadoop/hadoop-1.0.4/logs/history$ ls done/version-1/localhost1385716167800/2013/11/29/000000/ job_201311291709_0001_1385739836098_libenchao_word+count job_201311291709_0001_conf.xml