search

suharev7 / clickhouse-rs

Asynchronous ClickHouse client library for Rust programming language.
MIT License
324 stars 121 forks source link

Expose the internal column data so that entire data can be copied directly #221

Open saherawat opened 4 months ago

saherawat commented 4 months ago

This change has helped us copying the data into apache arrow structures directly and it has improved the latency significantly (6 times faster) as compared to copying individual elements.

saherawat commented 4 months ago

@suharev7 can you please take a look at this PR?

saherawat commented 2 months ago

@suharev7 can you please take a look at this PR. Please help us get this checked in.

suharev7 commented 2 months ago

I just wonder how are you going to user ArcColumnData.

saherawat commented 2 months ago

We are using apache arrow and with this change we are able to directly copy the memory blocks into Apache arrow column's data.

suharev7 commented 2 months ago

I would like to be able to change the implementation of ColumnData trait in the future, so I don't want to expose this trait. I can add a new method that will consume Column and return Vec<T> without memory coping (when it's possible, for example, for primitive date types). Might this be suitable for you?