oap-project / gazelle_plugin

Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.
Apache License 2.0
256 stars 77 forks source link

Automatic control value num in buffers #1137

Open jackylee-ch opened 1 year ago

jackylee-ch commented 1 year ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. In WSCG, currently we use batchSize to control the values num in buffers. However, if we deal with large strings, it may lead to buffer overflows as failed to append string value to buffer.

Describe alternatives you've considered We can hold the exception when we failed to append value to buffer and stop early as buffer has already full.