NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
822 stars 235 forks source link

Remove batch size bytes limits #11746

Open zpuller opened 12 hours ago

zpuller commented 12 hours ago

Testing with higher batch sizes has shown potential performance improvements in certain scenarios, and has not shown any evidence of query failures or data corruption, so we think we are comfortable removing the existing limits we've been imposing on these configs.