numMidDimSplit 최적값 찾기

[namnam21]

computation overhead와 network overhead의 일반화

[kmu-leeky]

Left matrix: 1-4, Right matrix: 4-1. Each block size - 2048-512

• no midDimSplit - total: 8초, 64MB shuffle read, 32MB shuffle write, multiplication task: 8초 (http://bd-2.cs.kookmin.ac.kr:28080/history/application_1495847540145_0062/jobs/job/?id=9)
• 4 midDimsplit - total: 4초, 16MB shuffle read/32MB shuffle write (each task). total of 4 tasks, multiplication time per task: 2초, + final aggregation - shuffle read 128 MB - 1초 (http://bd-2.cs.kookmin.ac.kr:28080/history/application_1495847540145_0062/jobs/job/?id=14)

[kmu-leeky]

Left matrix: 1-32, Right matrix: 32-1 - Each block size 2048-64

• no midDimSplit - total: 15초 커널 13초. read 64M, write: 32MB. http://bd-2.cs.kookmin.ac.kr:28080/history/application_1495847540145_0064/jobs/job/?id=4
• 4 midDimSplit - total 7초 커널 6초, read 16M, Write 32M, final shuffle read - 128MB http://bd-2.cs.kookmin.ac.kr:28080/history/application_1495847540145_0064/jobs/job/?id=14
• 16 midDimSplit - total 8초 커널 2초, read 4M, write 32M, final shuffle read - 512MB http://bd-2.cs.kookmin.ac.kr:28080/history/application_1495847540145_0064/jobs/job/?id=19
  • 32 midDimSplit - total 11초 커널 1초, read 2M, write 32M, final shuffle read - 1G http://bd-2.cs.kookmin.ac.kr:28080/history/application_1495847540145_0064/jobs/job/?id=24

[kmu-leeky]

간단한 실험에 의하면 고려할 행렬 특성에 따라 고려할 요소는 shuffle read - 연산을 위해 연산 시간 shuffle write - 마지막 aggregation 정도 인듯