BulkCopy Remove MemoryStream To Reduce Memory Usage

DarkWanderer / ClickHouse.Client

.NET client for ClickHouse

MIT License

302 stars 58 forks source link

BulkCopy Remove MemoryStream To Reduce Memory Usage #464

Closed MJEdwin closed 2 months ago

MJEdwin commented 2 months ago

BulkCopy SerializeToBatch Use MemoryStream will cost lot of memory，this operation can be deferred to HttpContent to avoid memory allocation of MemoryStream

DarkWanderer commented 2 months ago

Hi. Thanks for your contribution - will take a look this week

DarkWanderer commented 2 months ago

Excellent idea, few minor notes:

It looks like BulkCopyHttpContent and Batch classes can be merged - there's not much sense in maintaining separate objects
I don't think it has merit to maintain the uncompressed path - there's essentially no circumstances where it outperforms CompressionLevel.Fastest
I think BulkCopyHttpContent still needs to be disposed in SendBatchAsync?

MJEdwin commented 2 months ago

Excellent idea, few minor notes:

It looks like BulkCopyHttpContent and Batch classes can be merged - there's not much sense in maintaining separate objects

I don't think it has merit to maintain the uncompressed path - there's essentially no circumstances where it outperforms CompressionLevel.Fastest

I think BulkCopyHttpContent still needs to be disposed in SendBatchAsync?

@DarkWanderer i solved it

DarkWanderer commented 2 months ago

Thank you for your contribution!