databendlabs / databend

𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
https://docs.databend.com
Other
7.87k stars 751 forks source link

bug: Slow individual insert #16886

Closed k-bx closed 1 day ago

k-bx commented 1 day ago

Search before asking

Version

nightly (2024-11-17)

What's Wrong?

I am trying out Databend in its simplest form: local Docker container (minio + databend). The experience is good so far, but there's a problem I've faced already. I am writing IOT device data to databend table as soon as I receive it, and performance for individual insert is too slow at the moment. It can't do even 100 insert operations in one second.

I can implement client-side buffering and do COPY FROM operation (or a bulk-request) on client side, but I'd love if Databend to be able to do this for me instead, so that I don't have to complicate my app's logic.

Maybe something like this is already present?

How to Reproduce?

Write a Rust program doing INSERT statement in a loop for 100000 items.

Are you willing to submit PR?

k-bx commented 1 day ago

Closing in favor of a question https://github.com/databendlabs/databend/discussions/16887