NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
56 stars 38 forks source link

[BUG] Recommend batch size bytes to 1GB #1399

Open kuhushukla opened 4 weeks ago

kuhushukla commented 4 weeks ago

Describe the bug Currently we try and recommend a batch size that could be more than the 1GB value which seems to work best for most customers. We should default to this for now at least for the first GPU run (when profiler runs on CPU logs)