search

sunliwen / poco

Poco v1.6
0 stars 1 forks source link

[Batch]tac操作需要2个小时，并有计算失败 #105

Open sunliwen opened 9 years ago

sunliwen commented 9 years ago

计算3775万条记录 2014-12-23 04:55:31|INFO|Backfiller|Count: 37750000, 3274.72780741 rows/sec

tac操作需要2个多小时。

2014-12-23 04:55:31|INFO|Batch Server|Execute /usr/bin/tac </cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/preprocessing/reversed_backfilled_raw_logs >/cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/preprocessing/backfilled_raw_logs
2014-12-23 07:05:24|INFO|Batch Server|Job Succ: PreprocessingFlow.do_reverse_reversed_backfilled_raw_logs
014-12-23 08:26:38|INFO|emit_cooccurances|One User Skipped: id=ANO_8526f5ba-3e51-4af7-aec8-2069eef5577a len(item_ids)=1168
2014-12-23 08:26:42|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_emit_cooccurances
2014-12-23 08:26:42|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_sort_cooccurances
2014-12-23 08:26:42|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_not_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_sorted
2014-12-23 08:26:45|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_sort_cooccurances
2014-12-23 08:26:45|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_count_cooccurances
2014-12-23 08:26:45|INFO|Batch Server|Execute uniq -c /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_counts_raw
2014-12-23 08:26:47|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_count_cooccurances
2014-12-23 08:26:47|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:26:54|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:26:54|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_calc_item_similarities
2014-12-23 08:27:21|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_calc_item_similarities
2014-12-23 08:27:21|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:27:23|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:27:23|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:27:23|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/item_similarities_bi_directional > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/item_similarities_bi_directional_sorted
2014-12-23 08:27:24|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:27:24|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_extract_top_n
2014-12-23 08:27:38|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_extract_top_n
2014-12-23 08:27:38|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:27:39|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:27:40|INFO|root|FlowEnd: VSimiliarityCalcFlow
2014-12-23 08:27:40|INFO|root|FlowBegin: PLOSimilarityCalcFlow
2014-12-23 08:27:40|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_extract_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_extract_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Execute sort -T /cube/services/batch/temp < /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_maybe_dup | uniq > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_sort_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_sorted
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_sort_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_calc_item_prefer_count
2014-12-23 08:41:52|INFO|Batch Server|Execute cut -d , -f 2 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_sorted | sort -T /cube/services/batch/temp | uniq -c > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/item_prefer_count
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_calc_item_prefer_count
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_calc_user_count
2014-12-23 08:41:52|INFO|Batch Server|Execute cut -d , -f 1 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_sorted | uniq | wc -l > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_count
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_calc_user_count
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_emit_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_emit_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_sort_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_not_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_sorted
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_sort_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_count_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Execute uniq -c /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_counts_raw
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_count_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_calc_item_similarities
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_calc_item_similarities
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/item_similarities_bi_directional > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/item_similarities_bi_directional_sorted
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_extract_top_n
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_extract_top_n
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_upload_item_similarities_result
                                                                                                                                                                                                                                                                                                                                                                                                                                                                         3804,1        98%
2014-12-23 08:11:02|INFO|HiveBased|HIVE_START calc_unique_sku
2014-12-23 08:11:14|INFO|HiveBased|HIVE_END calc_unique_sku
2014-12-23 08:11:14|INFO|HiveBased|HIVE_START calc_avg_item_amount
2014-12-23 08:11:26|INFO|HiveBased|HIVE_END calc_avg_item_amount
2014-12-23 08:11:26|INFO|HiveBased|HIVE_START calc_click_rec_buy
2014-12-23 08:11:53|CRITICAL|Batch Server|An Exception happened while running Job: <bound method HiveBasedStatisticsFlow.do_hive_based_calculations of <__main__.HiveBasedStatisticsFlow instance at 0x7f8b2dac7c68>>
Traceback (most recent call last):
  File "/cube/apps/poco/poco/services/batch/server.py", line 155, in _execJob
    callable()
  File "/cube/apps/poco/poco/services/batch/server.py", line 202, in do_hive_based_calculations
    connection, SITE_ID, self.getWorkDir(), backfilled_raw_logs_path)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 633, in hive_based_calculations
    do_calculations(connection, site_id, work_dir, backfilled_raw_logs_path, client)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 615, in do_calculations
    calc_click_rec_buy(site_id, connection, client)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 52, in wrapped_function
    result = function(*arg, **kws)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 453, in calc_click_rec_buy
    client.execute("INSERT OVERWRITE TABLE rec_buy "
  File "pylib/hive_service/ThriftHive.py", line 63, in execute
    self.recv_execute()
  File "pylib/hive_service/ThriftHive.py", line 84, in recv_execute
    raise result.ex
HiveServerException: HiveServerException(errorCode=9, message='Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask', SQLState='08S01')
2014-12-23 08:11:53|INFO|root|FlowEnd: HiveBasedStatisticsFlow
2014-12-23 08:11:53|INFO|root|FlowBegin: VSimiliarityCalcFlow
2014-12-23 08:11:53|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_extract_user_item_matrix
2014-12-23 08:26:27|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_extract_user_item_matrix
2014-12-23 08:26:27|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:26:27|INFO|Batch Server|Execute sort -T /cube/services/batch/temp < /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_item_matrix_maybe_dup | uniq > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_item_matrix
2014-12-23 08:26:29|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:26:29|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_sort_user_item_matrix
2014-12-23 08:26:29|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_item_matrix > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_item_matrix_sorted
2014-12-23 08:26:29|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_sort_user_item_matrix
2014-12-23 08:26:29|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_calc_item_prefer_count
2014-12-23 08:26:29|INFO|Batch Server|Execute cut -d , -f 2 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_item_matrix_sorted | sort -T /cube/services/batch/temp | uniq -c > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/item_prefer_count
2014-12-23 08:26:30|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_calc_item_prefer_count
2014-12-23 08:26:30|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_calc_user_count
2014-12-23 08:26:30|INFO|Batch Server|Execute cut -d , -f 1 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_item_matrix_sorted | uniq | wc -l > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/user_count
2014-12-23 08:26:31|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_calc_user_count
2014-12-23 08:26:31|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_emit_cooccurances
2014-12-23 08:26:31|INFO|emit_cooccurances|One User Skipped: id=101193 len(item_ids)=1472
2014-12-23 08:26:31|INFO|emit_cooccurances|One User Skipped: id=101410 len(item_ids)=1589
2014-12-23 08:26:31|INFO|emit_cooccurances|One User Skipped: id=105817 len(item_ids)=1036
2014-12-23 08:26:31|INFO|emit_cooccurances|One User Skipped: id=106100 len(item_ids)=2590
2014-12-23 08:26:31|INFO|emit_cooccurances|One User Skipped: id=106744 len(item_ids)=1647
2014-12-23 08:26:31|INFO|emit_cooccurances|One User Skipped: id=112080 len(item_ids)=2387
2014-12-23 08:26:32|INFO|emit_cooccurances|One User Skipped: id=153653 len(item_ids)=883
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=76287 len(item_ids)=2230
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=77389 len(item_ids)=867
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=77393 len(item_ids)=1525
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=77487 len(item_ids)=811
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=77494 len(item_ids)=851
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=77553 len(item_ids)=1545
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=78676 len(item_ids)=1564
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=80755 len(item_ids)=2306
2014-12-23 08:26:33|INFO|emit_cooccurances|One User Skipped: id=81855 len(item_ids)=1928
2014-12-23 08:26:34|INFO|emit_cooccurances|One User Skipped: id=ANO_09008ad5-a342-4f65-8128-2b4e78faf3d2 len(item_ids)=2485
2014-12-23 08:26:36|INFO|emit_cooccurances|One User Skipped: id=ANO_3e6513a2-c9c9-4d2f-9cd8-54be46ffeee9 len(item_ids)=930
2014-12-23 08:26:37|INFO|emit_cooccurances|One User Skipped: id=ANO_70623a3b-1525-460b-8415-e84dcae68c95 len(item_ids)=896
2014-12-23 08:26:38|INFO|emit_cooccurances|One User Skipped: id=ANO_8526f5ba-3e51-4af7-aec8-2069eef5577a len(item_ids)=1168
2014-12-23 08:26:42|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_emit_cooccurances
2014-12-23 08:26:42|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_sort_cooccurances
2014-12-23 08:26:42|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_not_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_sorted
2014-12-23 08:26:45|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_sort_cooccurances
2014-12-23 08:26:45|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_count_cooccurances
2014-12-23 08:26:45|INFO|Batch Server|Execute uniq -c /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/cooccurances_counts_raw
2014-12-23 08:26:47|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_count_cooccurances
2014-12-23 08:26:47|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:26:54|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:26:54|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_calc_item_similarities
2014-12-23 08:27:21|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_calc_item_similarities
2014-12-23 08:27:21|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:27:23|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:27:23|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:27:23|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/item_similarities_bi_directional > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:V/item_similarities_bi_directional_sorted
2014-12-23 08:27:24|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:27:24|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_extract_top_n
2014-12-23 08:27:38|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_extract_top_n
2014-12-23 08:27:38|INFO|Batch Server|Start Job: VSimiliarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:27:39|INFO|Batch Server|Job Succ: VSimiliarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:27:40|INFO|root|FlowEnd: VSimiliarityCalcFlow
2014-12-23 08:27:40|INFO|root|FlowBegin: PLOSimilarityCalcFlow
2014-12-23 08:27:40|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_extract_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_extract_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Execute sort -T /cube/services/batch/temp < /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_maybe_dup | uniq > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_sort_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_sorted
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_sort_user_item_matrix
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_calc_item_prefer_count
2014-12-23 08:41:52|INFO|Batch Server|Execute cut -d , -f 2 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_sorted | sort -T /cube/services/batch/temp | uniq -c > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/item_prefer_count
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_calc_item_prefer_count
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_calc_user_count
2014-12-23 08:41:52|INFO|Batch Server|Execute cut -d , -f 1 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_item_matrix_sorted | uniq | wc -l > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/user_count
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_calc_user_count
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_emit_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_emit_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_sort_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_not_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_sorted
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_sort_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_count_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Execute uniq -c /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/cooccurances_counts_raw
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_count_cooccurances
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:41:52|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_format_cooccurances_counts
2014-12-23 08:41:52|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_calc_item_similarities
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_calc_item_similarities
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_make_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/item_similarities_bi_directional > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:PLO/item_similarities_bi_directional_sorted
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_extract_top_n
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_extract_top_n
2014-12-23 08:41:53|INFO|Batch Server|Start Job: PLOSimilarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:41:53|INFO|Batch Server|Job Succ: PLOSimilarityCalcFlow.do_upload_item_similarities_result
2014-12-23 08:41:53|INFO|root|FlowEnd: PLOSimilarityCalcFlow
2014-12-23 08:41:53|INFO|root|FlowBegin: BuyTogetherSimilarityFlow
2014-12-23 08:41:53|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_extract_user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_extract_user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Execute sort -T /cube/services/batch/temp < /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_item_matrix_maybe_dup | uniq > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_de_duplicate_user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_sort_user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_item_matrix > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_item_matrix_sorted
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_sort_user_item_matrix
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_calc_item_prefer_count
2014-12-23 08:56:03|INFO|Batch Server|Execute cut -d , -f 2 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_item_matrix_sorted | sort -T /cube/services/batch/temp | uniq -c > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/item_prefer_count
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_calc_item_prefer_count
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_calc_user_count
2014-12-23 08:56:03|INFO|Batch Server|Execute cut -d , -f 1 /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_item_matrix_sorted | uniq | wc -l > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/user_count
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_calc_user_count
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_emit_cooccurances
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_emit_cooccurances
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_sort_cooccurances
2014-12-23 08:56:03|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/cooccurances_not_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/cooccurances_sorted
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_sort_cooccurances
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_count_cooccurances
2014-12-23 08:56:03|INFO|Batch Server|Execute uniq -c /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/cooccurances_sorted > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/cooccurances_counts_raw
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_count_cooccurances
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_format_cooccurances_counts
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_format_cooccurances_counts
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_calc_item_similarities
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_calc_item_similarities
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_make_item_similarities_bi_directional
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_make_item_similarities_bi_directional
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:56:03|INFO|Batch Server|Execute sort -T /cube/services/batch/temp /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/item_similarities_bi_directional > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/similarities-calc:BuyTogether/item_similarities_bi_directional_sorted
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_sort_item_similarities_bi_directional
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_extract_top_n
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_extract_top_n
2014-12-23 08:56:03|INFO|Batch Server|Start Job: BuyTogetherSimilarityFlow.do_upload_item_similarities_result
2014-12-23 08:56:03|INFO|Batch Server|Job Succ: BuyTogetherSimilarityFlow.do_upload_item_similarities_result
2014-12-23 08:56:04|INFO|root|FlowEnd: BuyTogetherSimilarityFlow
2014-12-23 08:56:04|INFO|root|FlowBegin: ViewedUltimatelyBuyFlow
2014-12-23 08:56:04|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.do_extract_user_view_buy_logs
2014-12-23 09:11:28|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.do_extract_user_view_buy_logs
2014-12-23 09:11:28|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.do_sort_user_view_buy_logs
2014-12-23 09:11:28|INFO|Batch Server|Execute sort -T /cube/services/batch/temp </cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/user_view_buy_logs >/cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/user_view_buy_logs_sorted
2014-12-23 09:11:29|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.do_sort_user_view_buy_logs
2014-12-23 09:11:29|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.do_pair_view_buy
2014-12-23 09:11:33|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.do_pair_view_buy
2014-12-23 09:11:33|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.count_pairs
2014-12-23 09:11:33|INFO|Batch Server|Execute sort -T /cube/services/batch/temp </cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/view_buy_pairs | uniq -c >/cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/view_buy_pairs_counted
2014-12-23 09:11:34|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.count_pairs
2014-12-23 09:11:34|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.do_extract_user_item_matrix
2014-12-23 09:26:25|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.do_extract_user_item_matrix
2014-12-23 09:26:25|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.do_de_duplicate_user_item_matrix
2014-12-23 09:26:25|INFO|Batch Server|Execute sort -T /cube/services/batch/temp < /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/user_item_matrix_maybe_dup | uniq > /cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/user_item_matrix
2014-12-23 09:26:27|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.do_de_duplicate_user_item_matrix
2014-12-23 09:26:27|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.count_item_view
2014-12-23 09:26:27|INFO|Batch Server|Execute cut -d , -f 2 </cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/user_item_matrix | sort -T /cube/services/batch/temp | uniq -c >/cube/data/poco/batch/work_dir/haoyaoshi/20141223_014324_ed3e2c69-9446-4bb3-b62c-2ffdfb14c938/ViewedUltimatelyBuy/item_view_times
2014-12-23 09:26:28|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.count_item_view
2014-12-23 09:26:28|INFO|Batch Server|Start Job: ViewedUltimatelyBuyFlow.upload_viewed_ultimately_buy
2014-12-23 09:26:32|INFO|Batch Server|Job Succ: ViewedUltimatelyBuyFlow.upload_viewed_ultimately_buy
2014-12-23 09:26:32|INFO|root|FlowEnd: ViewedUltimatelyBuyFlow
2014-12-23 09:26:32|INFO|Batch Server|END CALCULATION ON:haoyaoshi, RESULT:FAIL, CALCULATION_ID:ed3e2c69-9446-4bb3-b62c-2ffdfb14c938

有一条计算还是失败，但是看起来不是critical的计算。

sunliwen commented 9 years ago

这还是一个很重要的计算，将时间跨度调整到30天重新开始计算，看是否与数据量相关。

sunliwen commented 9 years ago

15天得计算周期，就不会有失败了。 看来下面这个hive脚本还是有问题。

2014-12-23 08:11:26|INFO|HiveBased|HIVE_START calc_click_rec_buy
2014-12-23 08:11:53|CRITICAL|Batch Server|An Exception happened while running Job: <bound method HiveBasedStatisticsFlow.do_hive_based_calculations of <__main__.HiveBasedStatisticsFlow instance at 0x7f8b2dac7c68>>
Traceback (most recent call last):
  File "/cube/apps/poco/poco/services/batch/server.py", line 155, in _execJob
    callable()
  File "/cube/apps/poco/poco/services/batch/server.py", line 202, in do_hive_based_calculations
    connection, SITE_ID, self.getWorkDir(), backfilled_raw_logs_path)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 633, in hive_based_calculations
    do_calculations(connection, site_id, work_dir, backfilled_raw_logs_path, client)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 615, in do_calculations
    calc_click_rec_buy(site_id, connection, client)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 52, in wrapped_function
    result = function(*arg, **kws)
  File "/home/pocoweb/cube/apps/poco/poco/services/batch/statistics/hive_based_calculations.py", line 453, in calc_click_rec_buy
    client.execute("INSERT OVERWRITE TABLE rec_buy "
  File "pylib/hive_service/ThriftHive.py", line 63, in execute
    self.recv_execute()
  File "pylib/hive_service/ThriftHive.py", line 84, in recv_execute
    raise result.ex
HiveServerException: HiveServerException(errorCode=9, message='Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask', SQLState='08S01')
2014-12-23 08:11:53|INFO|root|FlowEnd: HiveBasedStatisticsFlow