search

Azure / logstash-output-kusto

Logstash output for Kusto
Apache License 2.0
13 stars 14 forks source link

Set max_threads #46

Closed Christodde closed 1 year ago

Christodde commented 1 year ago

Hi I have seen that the ingestor class is using max_threads which equals to 8.

Would it be possible to pass this parameter in the kusto plugin?;eg max_threads => 16

I have seen that as per official documentation by Microsoft ( https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-logstash ) this number of threads cannot be changed to any other value. I was wondering if I can use this parameter in kusto plugin despite that documentation does not refer to it it

ag-ramachandran commented 1 year ago

Hello Christodde,

Here are some options we can try. Will also use this issue to update github docs as well.

upload_concurrent_count -> controls the parallel threads created , so here probably we can increase the threads working on the upload. We can use this to increase the threads running in parallel. This will override the value 8 we have.

upload_queue_size: is also a value we can use. This is number of uploads that get queued before processing. Default is 30 so this is good to start but can be tuned as well

Christodde commented 1 year ago

Many thanks for your prompt response We have tried the upload_concurrent_count and it works fine.