aljazerzen / connector_arrow

Apache Arrow database client for many databases.
https://docs.rs/connector_arrow
MIT License
37 stars 2 forks source link

perf: transport should do branching ahead of the hot path #9

Open aljazerzen opened 6 months ago

aljazerzen commented 6 months ago

When moving values from a producer to a consumer, we utilize crate::utils::transport::transport function.

It contains branching depending on the type of the field that is being transported. In most of the cases, we iterate over an array (or multiple arrays), which means that fields repeat a lot.

It would (probably) be much more performant to have a transporter function instead, that would be specialized for a specific type. These functions would be created prior to iteration over the arrays.

In the case of converting rows into RecordBatch, we could have these functions in an array.