search

hortonworks-spark / spark-llap

Apache License 2.0
101 stars 68 forks source link

Difference in resource requests via spark-llap #283

Open John-yovoa opened 4 years ago

John-yovoa commented 4 years ago

Query: select DISTINCT student_id from mytestdb.mytesttable;

From TEZ mode only 2 reducers are required but with LLAP it requests 532 reducers

from TEZ:

VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED

Map 1 .......... container SUCCEEDED 95 95 0 0 0 0 Reducer 2 ...... container SUCCEEDED 2 2 0 0 0 0

VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 19.95 s

INFO : Query Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : OPERATION DURATION INFO : ---------------------------------------------------------------------------------------------- INFO : Compile Query 0.72s INFO : Prepare Plan 7.20s INFO : Get Query Coordinator (AM) 0.00s INFO : Submit Plan 0.23s INFO : Start DAG 0.81s INFO : Run DAG 19.91s INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : Task Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 12119.00 1,251,490 37,190 819,833,947 15,665 INFO : Reducer 2 7747.00 7,150 620 15,665 0 INFO : ---------------------------------------------------------

From LLAP:

VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED

Map 1 .......... llap SUCCEEDED 96 96 0 0 0 0 Reducer 2 ...... llap SUCCEEDED 532 532 0 0 0 2

VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 29.83 s

`INFO : Query Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : OPERATION DURATION INFO : ---------------------------------------------------------------------------------------------- INFO : Compile Query 11.87s INFO : Prepare Plan 0.07s INFO : Get Query Coordinator (AM) 0.00s INFO : Submit Plan 0.03s INFO : Start DAG 0.53s INFO : Run DAG 29.77s INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : Task Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 27782.00 0 0 819,833,947 15,216 INFO : Reducer 2 24735.00 0 0 15,216 0 INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : LLAP IO Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS TOTAL_IO INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 83209 4450 524 384.62MB 47.76MB 781.92s

INFO : INFO : FileSystem Counters Summary INFO : INFO : Scheme: FILE INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 2.45MB 0 0 1.85MB 0 INFO : Reducer 2 0B 0 0 0B 0 INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : Scheme: HDFS INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 54.68MB 1048 0 0B 0 INFO : Reducer 2 0B 615 0 25.77KB 410 INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : LLAP WM Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES SPECULATIVE QUEUED SPECULATIVE RUNNING GUARANTEED QUEUED GUARANTEED RUNNING INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 0.00s 0.00s 0.04s 1020.11s INFO : Reducer 2 38.81s 43.56s 192.68s 9882.59s INFO : ---------------------------------------------------------------------------------------------- INFO : INFO :

`