search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.18k stars 3.51k forks source link

Slow Data ingestion in influxdb #24915

Open Satish2007 opened 2 months ago

Satish2007 commented 2 months ago

Hi all, Facing peculiar slow data ingestion issue. H/w and s/w Info: 16vCPU 32GB RAM 100GB Default configuration of config file Engine type: TSM Version: 1.8.10 OSS Community OS : Ubuntu 22.04

Issue can be reproduce, we tried to test performance in test setup. We are just writing data points in batch size of 250/sec. Only one datastore, 1200 measurements, 6000 series cardinality.

We are generating Load thru c# application. When we start c# application to ingest data , for first few batches it will be able to ingest 250 batch size in mili seconds and then gradually write time increases to minutes.

Everytime same data is being inserted.

No high CPU , RAM or Disk IO being observed while testing the scenario. No write failed or timeout errors observed in log file.

It has become road blocker for us.

Please help/advice to solve the issue. Let me know if you need any details.

Thanks in advance @davidby-influx , @sanderson .

Regards

davidby-influx commented 2 weeks ago

My immediate suspicion is that this is related to your c# program. Can you replicate this with the inch tool?