Closed res-life closed 1 year ago
It's not related to file cache
.
It also occurs when file cache
is disabled and Alluxio is enabled.
Also occurs at query9.
We increased the memory pressure in some places when trying to add in retry work. #7672 should fix the issue and is currently being worked on.
I confirmed that this is reproducible without Alluxio and without file caching enabled.
databricks by default is using cuda 11.4 so we don't have the async allocator there. I have seen OOM's sometimes in the past due to fragementation.
I validated that the i3.2xlarge + 4 * g4dn.4xlarge
NDS SF3K benchmark will succeed without any query failures with a test jar with this fix: https://github.com/NVIDIA/spark-rapids/issues/7672
Validated on latest snapshot jar with NDS SF3K benchmark with i3.2xlarge + 4 * g4dn.4xlarge
cluster that no query failures exist.
Results:
====== Run query96 ======
TaskFailureListener is registered.
Time taken: [19880] millis for query96
====== Run query7 ======
TaskFailureListener is registered.
Time taken: [12176] millis for query7
====== Run query75 ======
TaskFailureListener is registered.
Time taken: [49120] millis for query75
====== Run query44 ======
TaskFailureListener is registered.
Time taken: [4414] millis for query44
====== Run query39_part1 ======
TaskFailureListener is registered.
Time taken: [5818] millis for query39_part1
====== Run query39_part2 ======
TaskFailureListener is registered.
Time taken: [3724] millis for query39_part2
====== Run query80 ======
TaskFailureListener is registered.
Time taken: [25905] millis for query80
====== Run query32 ======
TaskFailureListener is registered.
Time taken: [4189] millis for query32
====== Run query19 ======
TaskFailureListener is registered.
Time taken: [4544] millis for query19
====== Run query25 ======
TaskFailureListener is registered.
Time taken: [7889] millis for query25
====== Run query78 ======
TaskFailureListener is registered.
Time taken: [106146] millis for query78
====== Run query86 ======
TaskFailureListener is registered.
Time taken: [3474] millis for query86
====== Run query1 ======
TaskFailureListener is registered.
Time taken: [5370] millis for query1
====== Run query91 ======
TaskFailureListener is registered.
Time taken: [3278] millis for query91
====== Run query21 ======
TaskFailureListener is registered.
Time taken: [1612] millis for query21
====== Run query43 ======
TaskFailureListener is registered.
Time taken: [4416] millis for query43
====== Run query27 ======
TaskFailureListener is registered.
Time taken: [9193] millis for query27
====== Run query94 ======
TaskFailureListener is registered.
Time taken: [23573] millis for query94
====== Run query45 ======
TaskFailureListener is registered.
Time taken: [2857] millis for query45
====== Run query58 ======
TaskFailureListener is registered.
Time taken: [2903] millis for query58
====== Run query64 ======
TaskFailureListener is registered.
Time taken: [78379] millis for query64
====== Run query36 ======
TaskFailureListener is registered.
Time taken: [7803] millis for query36
====== Run query33 ======
TaskFailureListener is registered.
Time taken: [4199] millis for query33
====== Run query46 ======
TaskFailureListener is registered.
Time taken: [9450] millis for query46
====== Run query62 ======
TaskFailureListener is registered.
Time taken: [8595] millis for query62
====== Run query16 ======
TaskFailureListener is registered.
Time taken: [4460] millis for query16
====== Run query10 ======
TaskFailureListener is registered.
Time taken: [5022] millis for query10
====== Run query63 ======
TaskFailureListener is registered.
Time taken: [5986] millis for query63
====== Run query69 ======
TaskFailureListener is registered.
Time taken: [3991] millis for query69
====== Run query60 ======
TaskFailureListener is registered.
Time taken: [5732] millis for query60
====== Run query59 ======
TaskFailureListener is registered.
Time taken: [15394] millis for query59
====== Run query37 ======
TaskFailureListener is registered.
Time taken: [8350] millis for query37
====== Run query98 ======
TaskFailureListener is registered.
Time taken: [3504] millis for query98
====== Run query85 ======
TaskFailureListener is registered.
Time taken: [10754] millis for query85
====== Run query70 ======
TaskFailureListener is registered.
Time taken: [9281] millis for query70
====== Run query67 ======
TaskFailureListener is registered.
Time taken: [647998] millis for query67
====== Run query28 ======
TaskFailureListener is registered.
Time taken: [80697] millis for query28
====== Run query81 ======
TaskFailureListener is registered.
Time taken: [6388] millis for query81
====== Run query97 ======
TaskFailureListener is registered.
Time taken: [15391] millis for query97
====== Run query66 ======
TaskFailureListener is registered.
Time taken: [12784] millis for query66
====== Run query90 ======
TaskFailureListener is registered.
Time taken: [6108] millis for query90
====== Run query17 ======
TaskFailureListener is registered.
Time taken: [9418] millis for query17
====== Run query47 ======
TaskFailureListener is registered.
Time taken: [11350] millis for query47
====== Run query95 ======
TaskFailureListener is registered.
Time taken: [38455] millis for query95
====== Run query92 ======
TaskFailureListener is registered.
Time taken: [2664] millis for query92
====== Run query3 ======
TaskFailureListener is registered.
Time taken: [7839] millis for query3
====== Run query51 ======
TaskFailureListener is registered.
Time taken: [7959] millis for query51
====== Run query35 ======
TaskFailureListener is registered.
Time taken: [7043] millis for query35
====== Run query49 ======
TaskFailureListener is registered.
Time taken: [16491] millis for query49
====== Run query9 ======
TaskFailureListener is registered.
Time taken: [21549] millis for query9
====== Run query31 ======
TaskFailureListener is registered.
Time taken: [12092] millis for query31
====== Run query11 ======
TaskFailureListener is registered.
Time taken: [33898] millis for query11
====== Run query93 ======
TaskFailureListener is registered.
Time taken: [229439] millis for query93
====== Run query29 ======
TaskFailureListener is registered.
Time taken: [20076] millis for query29
====== Run query38 ======
TaskFailureListener is registered.
Time taken: [11556] millis for query38
====== Run query22 ======
TaskFailureListener is registered.
Time taken: [3954] millis for query22
====== Run query89 ======
TaskFailureListener is registered.
Time taken: [5318] millis for query89
====== Run query15 ======
TaskFailureListener is registered.
Time taken: [3497] millis for query15
====== Run query6 ======
TaskFailureListener is registered.
Time taken: [2011] millis for query6
====== Run query52 ======
TaskFailureListener is registered.
Time taken: [4053] millis for query52
====== Run query50 ======
TaskFailureListener is registered.
Time taken: [140514] millis for query50
====== Run query42 ======
TaskFailureListener is registered.
Time taken: [2867] millis for query42
====== Run query41 ======
TaskFailureListener is registered.
Time taken: [816] millis for query41
====== Run query8 ======
TaskFailureListener is registered.
Time taken: [3618] millis for query8
====== Run query12 ======
TaskFailureListener is registered.
Time taken: [1881] millis for query12
====== Run query20 ======
TaskFailureListener is registered.
Time taken: [1950] millis for query20
====== Run query88 ======
TaskFailureListener is registered.
Time taken: [66945] millis for query88
====== Run query82 ======
TaskFailureListener is registered.
Time taken: [14500] millis for query82
====== Run query23_part1 ======
TaskFailureListener is registered.
Time taken: [48910] millis for query23_part1
====== Run query23_part2 ======
TaskFailureListener is registered.
Time taken: [48231] millis for query23_part2
====== Run query14_part1 ======
TaskFailureListener is registered.
Time taken: [30410] millis for query14_part1
====== Run query14_part2 ======
TaskFailureListener is registered.
Time taken: [24861] millis for query14_part2
====== Run query57 ======
TaskFailureListener is registered.
Time taken: [7074] millis for query57
====== Run query65 ======
TaskFailureListener is registered.
Time taken: [31166] millis for query65
====== Run query71 ======
TaskFailureListener is registered.
Time taken: [9616] millis for query71
====== Run query34 ======
TaskFailureListener is registered.
Time taken: [4916] millis for query34
====== Run query48 ======
TaskFailureListener is registered.
Time taken: [6922] millis for query48
====== Run query30 ======
TaskFailureListener is registered.
Time taken: [6529] millis for query30
====== Run query74 ======
TaskFailureListener is registered.
Time taken: [22272] millis for query74
====== Run query87 ======
TaskFailureListener is registered.
Time taken: [12607] millis for query87
====== Run query77 ======
TaskFailureListener is registered.
Time taken: [7064] millis for query77
====== Run query73 ======
TaskFailureListener is registered.
Time taken: [3262] millis for query73
====== Run query84 ======
TaskFailureListener is registered.
Time taken: [4278] millis for query84
====== Run query54 ======
TaskFailureListener is registered.
Time taken: [5747] millis for query54
====== Run query55 ======
TaskFailureListener is registered.
Time taken: [2622] millis for query55
====== Run query56 ======
TaskFailureListener is registered.
Time taken: [4163] millis for query56
====== Run query2 ======
TaskFailureListener is registered.
Time taken: [13934] millis for query2
====== Run query26 ======
TaskFailureListener is registered.
Time taken: [6269] millis for query26
====== Run query40 ======
TaskFailureListener is registered.
Time taken: [11906] millis for query40
====== Run query72 ======
TaskFailureListener is registered.
Time taken: [39690] millis for query72
====== Run query53 ======
TaskFailureListener is registered.
Time taken: [4198] millis for query53
====== Run query79 ======
TaskFailureListener is registered.
Time taken: [8341] millis for query79
====== Run query18 ======
TaskFailureListener is registered.
Time taken: [11206] millis for query18
====== Run query13 ======
TaskFailureListener is registered.
Time taken: [9851] millis for query13
====== Run query24_part1 ======
TaskFailureListener is registered.
Time taken: [52977] millis for query24_part1
====== Run query24_part2 ======
TaskFailureListener is registered.
Time taken: [50473] millis for query24_part2
====== Run query4 ======
TaskFailureListener is registered.
Time taken: [100992] millis for query4
====== Run query99 ======
TaskFailureListener is registered.
Time taken: [16626] millis for query99
====== Run query68 ======
TaskFailureListener is registered.
Time taken: [5855] millis for query68
====== Run query83 ======
TaskFailureListener is registered.
Time taken: [2701] millis for query83
====== Run query61 ======
TaskFailureListener is registered.
Time taken: [5336] millis for query61
====== Run query5 ======
TaskFailureListener is registered.
Time taken: [8548] millis for query5
====== Run query76 ======
TaskFailureListener is registered.
Time taken: [26556] millis for query76
====== Power Test Time: 2690208 milliseconds ======
====== Total Time: 2858784 milliseconds ======
Describe the bug Hit GPU OOM when running NDS power run on Databricks.
Steps/Code to reproduce bug Git check branch-22.06; Compile DB 321 jar Start DB cluster with
i3.2xlarge + 4 * g4dn.4xlarge
instances Configurations are:Docker is: gaochong365/rapids-4-spark-databricks:23.04.0-rc1 Zone is us-west-2a Open an notebook and run NDS:
Environment details (please complete the following information)
Additional context DB version is: 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12) Details logs: