Open nhtha opened 2 years ago
Another error is: "ERROR Operation UNLOAD_20221111-085837-498980 interrupted, waiting for termination."
Replaying from checkpoint file: /data/ptdl_qlrr/logs/UNLOAD_20221111-080630-863209/checkpoint.csv. Record metrics will reflect totals from the previous operation; other metrics won't be affected. Only errors and rejected records generated by the current operation will be reported in its log files. Username and password provided but auth provider not specified, inferring PlainTextAuthProvider Operation directory: /data/ptdl_qlrr/logs/UNLOAD_20221111-085837-498980 total | failed | rows/s | p50ms | p99ms | p999ms 3,467,407,673 | 0 | 621,193 | 95.97 | 497.03 | 1,317.01 total | failed | rows/s | p50ms | p99ms | p999ms 3,469,073,099 | 0 | 837,269 | 80.00 | 476.05 | 5,200.94 total | failed | rows/s | p50ms | p99ms | p999ms 3,470,757,832 | 0 | 882,329 | 73.42 | 478.15 | 1,317.01 total | failed | rows/s | p50ms | p99ms | p999ms 3,472,159,454 | 0 | 858,345 | 71.92 | 520.09 | 1,317.01 total | failed | rows/s | p50ms | p99ms | p999ms 3,474,032,109 | 0 | 876,488 | 68.21 | 476.05 | 1,266.68 total | failed | rows/s | p50ms | p99ms | p999ms 3,475,645,971 | 0 | 876,452 | 66.39 | 476.05 | 1,241.51 total | failed | rows/s | p50ms | p99ms | p999ms 3,477,578,159 | 0 | 872,182 | 66.52 | 478.15 | 1,249.90 total | failed | rows/s | p50ms | p99ms | p999ms 3,480,153,344 | 0 | 884,311 | 67.03 | 549.45 | 1,266.68 total | failed | rows/s | p50ms | p99ms | p999ms 3,482,625,042 | 0 | 890,819 | 68.06 | 599.79 | 1,342.18 total | failed | rows/s | p50ms | p99ms | p999ms 3,485,110,419 | 0 | 904,192 | 67.69 | 566.23 | 1,266.68 total | failed | rows/s | p50ms | p99ms | p999ms 3,487,414,102 | 0 | 915,475 | 67.42 | 549.45 | 1,317.01 total | failed | rows/s | p50ms | p99ms | p999ms 3,489,846,608 | 0 | 920,391 | 67.39 | 557.84 | 1,266.68 total | failed | rows/s | p50ms | p99ms | p999ms 3,492,289,412 | 0 | 927,757 | 67.93 | 566.23 | 1,375.73 total | failed | rows/s | p50ms | p99ms | p999ms 3,494,751,096 | 0 | 938,479 | 68.19 | 557.84 | 1,375.73 total | failed | rows/s | p50ms | p99ms | p999ms 3,496,945,802 | 0 | 939,778 | 69.40 | 566.23 | 1,375.73 total | failed | rows/s | p50ms | p99ms | p999ms 3,499,404,990 | 0 | 947,484 | 69.85 | 566.23 | 1,375.73 total | failed | rows/s | p50ms | p99ms | p999ms 3,502,007,109 | 0 | 947,507 | 70.18 | 578.81 | 1,367.34 total | failed | rows/s | p50ms | p99ms | p999ms 3,504,800,901 | 0 | 948,804 | 70.85 | 583.01 | 1,367.34 total | failed | rows/s | p50ms | p99ms | p999ms 3,507,452,677 | 0 | 948,107 | 71.72 | 603.98 | 1,375.73 total | failed | rows/s | p50ms | p99ms | p999ms 3,510,060,768 | 0 | 947,240 | 72.36 | 616.56 | 1,375.73 total | failed | rows/s | p50ms | p99ms | p999ms 3,512,574,984 | 0 | 943,762 | 73.60 | 624.95 | 1,426.06 total | failed | rows/s | p50ms | p99ms | p999ms 3,515,309,352 | 0 | 944,607 | 74.03 | 629.15 | 1,426.06 total | failed | rows/s | p50ms | p99ms | p999ms 3,517,811,955 | 0 | 945,014 | 74.06 | 629.15 | 1,434.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,520,392,326 | 0 | 947,958 | 73.90 | 633.34 | 1,434.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,523,118,045 | 0 | 950,198 | 73.61 | 629.15 | 1,442.84 total | failed | rows/s | p50ms | p99ms | p999ms 3,525,756,065 | 0 | 951,747 | 73.63 | 633.34 | 1,509.95 total | failed | rows/s | p50ms | p99ms | p999ms 3,528,408,314 | 0 | 950,694 | 73.71 | 637.53 | 1,535.12 total | failed | rows/s | p50ms | p99ms | p999ms 3,531,158,764 | 0 | 947,925 | 73.65 | 633.34 | 1,535.12 total | failed | rows/s | p50ms | p99ms | p999ms 3,533,852,647 | 0 | 948,780 | 73.68 | 637.53 | 1,543.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,536,404,171 | 0 | 943,654 | 73.61 | 645.92 | 1,543.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,539,208,190 | 0 | 940,558 | 73.30 | 645.92 | 1,543.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,542,280,983 | 0 | 939,520 | 72.86 | 641.73 | 1,543.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,545,373,635 | 0 | 938,960 | 72.32 | 633.34 | 1,543.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,548,419,923 | 0 | 936,456 | 72.11 | 633.34 | 1,543.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,551,439,484 | 0 | 933,308 | 71.98 | 637.53 | 1,551.89 total | failed | rows/s | p50ms | p99ms | p999ms 3,554,529,078 | 0 | 930,384 | 71.72 | 637.53 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,557,679,507 | 0 | 927,049 | 71.51 | 637.53 | 1,551.89 total | failed | rows/s | p50ms | p99ms | p999ms 3,560,582,403 | 0 | 922,707 | 71.59 | 645.92 | 1,551.89 total | failed | rows/s | p50ms | p99ms | p999ms 3,563,117,509 | 0 | 917,155 | 71.71 | 650.12 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,565,760,562 | 0 | 912,980 | 71.86 | 650.12 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,568,465,381 | 0 | 909,332 | 72.03 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,571,106,600 | 0 | 905,237 | 72.26 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,573,534,345 | 0 | 901,551 | 72.44 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,576,104,303 | 0 | 898,450 | 72.61 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,578,504,211 | 0 | 895,102 | 72.76 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,580,124,518 | 0 | 888,443 | 73.03 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,580,746,383 | 0 | 878,610 | 73.34 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,581,476,618 | 0 | 868,578 | 73.42 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,582,320,954 | 0 | 858,830 | 73.47 | 654.31 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,583,198,193 | 0 | 849,851 | 73.65 | 658.51 | 1,585.45 total | failed | rows/s | p50ms | p99ms | p999ms 3,584,227,992 | 0 | 840,575 | 74.00 | 658.51 | 1,593.84 total | failed | rows/s | p50ms | p99ms | p999ms 3,585,321,836 | 0 | 831,633 | 74.53 | 658.51 | 1,593.84 total | failed | rows/s | p50ms | p99ms | p999ms 3,586,427,840 | 0 | 823,299 | 75.01 | 658.51 | 1,602.22 total | failed | rows/s | p50ms | p99ms | p999ms 3,587,512,654 | 0 | 815,232 | 75.45 | 662.70 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,588,605,129 | 0 | 807,851 | 75.89 | 662.70 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,589,830,253 | 0 | 800,820 | 76.21 | 662.70 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,591,224,071 | 0 | 794,423 | 76.39 | 666.89 | 1,677.72 total | failed | rows/s | p50ms | p99ms | p999ms 3,593,176,587 | 0 | 789,792 | 76.55 | 671.09 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,595,580,233 | 0 | 787,426 | 76.62 | 671.09 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,598,033,509 | 0 | 785,015 | 76.62 | 671.09 | 1,677.72 total | failed | rows/s | p50ms | p99ms | p999ms 3,600,587,247 | 0 | 783,101 | 76.66 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,603,183,835 | 0 | 781,250 | 76.57 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,605,904,786 | 0 | 778,486 | 76.51 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,608,769,828 | 0 | 776,310 | 76.53 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,611,567,585 | 0 | 773,894 | 76.43 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,614,266,835 | 0 | 771,337 | 76.22 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,616,317,657 | 0 | 767,022 | 76.25 | 675.28 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,618,345,925 | 0 | 762,650 | 76.36 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,620,714,033 | 0 | 760,045 | 76.44 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,623,311,001 | 0 | 758,959 | 76.28 | 679.48 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,625,711,543 | 0 | 757,067 | 76.22 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,627,828,332 | 0 | 754,329 | 76.18 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,629,699,567 | 0 | 751,353 | 76.17 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,631,633,013 | 0 | 748,673 | 76.15 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,633,603,913 | 0 | 746,296 | 76.07 | 679.48 | 1,711.28 total | failed | rows/s | p50ms | p99ms | p999ms 3,635,541,213 | 0 | 743,810 | 75.96 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,637,514,586 | 0 | 741,293 | 75.85 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,639,341,598 | 0 | 738,462 | 75.81 | 679.48 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,640,745,137 | 0 | 735,049 | 75.84 | 675.28 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,642,247,749 | 0 | 732,202 | 75.86 | 675.28 | 1,694.50 total | failed | rows/s | p50ms | p99ms | p999ms 3,643,822,677 | 0 | 729,521 | 75.85 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,645,336,464 | 0 | 727,436 | 75.65 | 675.28 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,646,850,164 | 0 | 725,834 | 75.30 | 671.09 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,647,893,543 | 0 | 723,149 | 75.06 | 671.09 | 1,686.11 total | failed | rows/s | p50ms | p99ms | p999ms 3,648,640,147 | 0 | 720,434 | 74.88 | 671.09 | 1,677.72 total | failed | rows/s | p50ms | p99ms | p999ms 3,649,612,620 | 0 | 718,227 | 74.64 | 666.89 | 1,677.72 total | failed | rows/s | p50ms | p99ms | p999ms 3,650,586,656 | 0 | 716,754 | 74.34 | 662.70 | 1,635.78 total | failed | rows/s | p50ms | p99ms | p999ms 3,651,526,154 | 0 | 714,515 | 74.05 | 662.70 | 1,635.78 total | failed | rows/s | p50ms | p99ms | p999ms 3,652,396,230 | 0 | 712,141 | 73.78 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,653,346,670 | 0 | 708,998 | 73.59 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,654,581,132 | 0 | 705,902 | 73.38 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,655,937,382 | 0 | 703,333 | 73.08 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,657,387,667 | 0 | 700,620 | 72.88 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,659,056,942 | 0 | 698,273 | 72.73 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,660,628,630 | 0 | 696,440 | 72.75 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,662,293,804 | 0 | 695,115 | 72.80 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,664,057,864 | 0 | 693,905 | 72.80 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,665,701,051 | 0 | 692,422 | 72.79 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,667,303,787 | 0 | 690,678 | 72.75 | 650.12 | 1,610.61 total | failed | rows/s | p50ms | p99ms | p999ms 3,668,904,938 | 0 | 688,495 | 72.61 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,670,416,693 | 0 | 686,097 | 72.58 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,671,598,917 | 0 | 683,141 | 72.46 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,672,790,002 | 0 | 680,270 | 72.35 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,674,161,935 | 0 | 677,927 | 72.17 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,675,589,089 | 0 | 675,735 | 72.00 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,677,071,182 | 0 | 673,749 | 71.87 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,678,671,963 | 0 | 672,044 | 71.71 | 645.92 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,680,546,934 | 0 | 670,895 | 71.72 | 645.92 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,682,930,755 | 0 | 670,423 | 71.75 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,685,467,390 | 0 | 670,282 | 71.87 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,688,128,967 | 0 | 670,374 | 71.91 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,690,864,630 | 0 | 670,592 | 71.87 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,693,599,090 | 0 | 670,644 | 71.75 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,696,333,341 | 0 | 670,543 | 71.73 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,699,040,034 | 0 | 670,420 | 71.84 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,701,552,817 | 0 | 669,914 | 72.08 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,704,200,759 | 0 | 669,655 | 72.11 | 650.12 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,706,326,237 | 0 | 668,402 | 72.37 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,708,281,494 | 0 | 666,783 | 72.66 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,710,021,858 | 0 | 664,753 | 72.94 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,711,427,171 | 0 | 662,411 | 73.09 | 658.51 | 1,610.61 total | failed | rows/s | p50ms | p99ms | p999ms 3,712,810,301 | 0 | 659,973 | 73.11 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,714,124,534 | 0 | 657,470 | 73.05 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,715,411,398 | 0 | 655,028 | 72.98 | 658.51 | 1,610.61 total | failed | rows/s | p50ms | p99ms | p999ms 3,716,741,093 | 0 | 652,442 | 72.92 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,718,214,340 | 0 | 649,678 | 72.79 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,719,662,003 | 0 | 646,884 | 72.71 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,721,152,499 | 0 | 644,725 | 72.54 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,722,617,733 | 0 | 642,674 | 72.28 | 658.51 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,724,126,351 | 0 | 640,598 | 72.09 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,725,717,493 | 0 | 638,778 | 71.92 | 654.31 | 1,619.00 total | failed | rows/s | p50ms | p99ms | p999ms 3,727,132,699 | 0 | 636,774 | 71.75 | 654.31 | 1,610.61 Operation UNLOAD_20221111-085837-498980 interrupted, waiting for termination. total | failed | rows/s | p50ms | p99ms | p999ms 3,728,277,624 | 0 | 634,776 | 71.63 | 654.31 | 1,619.00 Operation UNLOAD_20221111-085837-498980 did not terminate within 10 seconds, forcing termination.
Dear dsbulk team, I have the following error when I use dsbulk 1.10 to unload data from DSE Analytics nodes: "Can't get more results because the continuous query has failed already. Most likely this is because the query was cancelled." Please help me to solve this. Many thanks, Ha Nguyen
2022-11-11 08:06:30 WARN Replaying from checkpoint file: /data/ptdl_qlrr/logs/UNLOAD_20221111-074333-416813/checkpoint.csv. 2022-11-11 08:06:30 WARN Record metrics will reflect totals from the previous operation; other metrics won't be affected. 2022-11-11 08:06:30 WARN Only errors and rejected records generated by the current operation will be reported in its log files. 2022-11-11 08:06:31 INFO Username and password provided but auth provider not specified, inferring PlainTextAuthProvider 2022-11-11 08:06:31 INFO Operation directory: /data/ptdl_qlrr/logs/UNLOAD_20221111-080630-863209 2022-11-11 08:48:32 ERROR Operation UNLOAD_20221111-080630-863209 failed: Can't get more results because the continuous query has failed already. Most likely this is because the query was cancelled. java.util.concurrent.CancellationException: Can't get more results because the continuous query has failed already. Most likely this is because the query was cancelled at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase$NodeResponseCallback.cancelledResultSetFuture(ContinuousRequestHandlerBase.java:1548) at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase$NodeResponseCallback.dequeueOrCreatePending(ContinuousRequestHandlerBase.java:1244) at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase.lambda$fetchNextPage$2(ContinuousRequestHandlerBase.java:306) at com.datastax.dse.driver.internal.core.cql.continuous.ContinuousRequestHandlerBase.fetchNextPage(ContinuousRequestHandlerBase.java:300) [3 skipped] at com.datastax.dse.driver.internal.core.cql.continuous.DefaultContinuousAsyncResultSet.fetchNextPage(DefaultContinuousAsyncResultSet.java:102) at com.datastax.oss.dsbulk.executor.api.subscription.ResultSubscription$Page.nextPage(ResultSubscription.java:585) at com.datastax.oss.dsbulk.executor.api.subscription.ResultSubscription.fetchNextPage(ResultSubscription.java:353) at com.datastax.oss.dsbulk.executor.api.subscription.ResultSubscription.lambda$fetchNextPage$3(ResultSubscription.java:392) at com.datastax.oss.dsbulk.executor.api.subscription.ResultSubscription.dequeue(ResultSubscription.java:450) [4 skipped] at com.datastax.oss.dsbulk.executor.api.subscription.ResultSubscription.tryNext(ResultSubscription.java:309) Suppressed: java.lang.Exception: #block terminated with an error at com.datastax.oss.dsbulk.workflow.unload.UnloadWorkflow.execute(UnloadWorkflow.java:208) [2 skipped] at com.datastax.oss.dsbulk.runner.WorkflowThread.run(WorkflowThread.java:53) 2022-11-11 08:58:34 INFO Final stats: 2022-11-11 08:58:34 INFO Errors are detailed in the following file(s): unload-errors.log 2022-11-11 08:58:34 INFO Checkpoints for the current operation were written to checkpoint.csv. 2022-11-11 08:58:34 INFO To resume the current operation, re-run it with the same settings, and add the following command line flag: 2022-11-11 08:58:34 INFO --dsbulk.log.checkpoint.file=/data/ptdl_qlrr/logs/UNLOAD_20221111-080630-863209/checkpoint.csv
┆Issue is synchronized with this Jira Task by Unito