search

NVIDIA / spark-rapids-benchmarks

Spark RAPIDS Benchmarks – benchmark sets and utilities for the RAPIDS Accelerator for Apache Spark
Apache License 2.0
36 stars 27 forks source link

Throughput run uses individual template in full benchmark #177

Closed wjxiz1992 closed 8 months ago

wjxiz1992 commented 8 months ago

close #174 As titled.

This PR adds in a new config for throughput run template file. This template file should have resource limit to allow concurrent streams during throughput test.

To avoid too many template files and also considering the highly overlap content between that and power run template, we don't adds in additional template files. User can copy the one for Power Run directly and add resource limit.

mattahrens commented 8 months ago

Do you want to add the sample throughput_*.template files in this PR as well so the examples are in the repo?

wjxiz1992 commented 8 months ago

No, in that way I'll add another 6 new template files( (cpu/gpu) *(pure parquet/delta/iceberg/) ) but with just like 1 or 2 lines diff for resource limits comparing to the Power template. That's why I mentioned in the README that user can create based on the Power ones.