search

Azure / azure-kusto-spark

Apache Spark Connector for Azure Kusto
Apache License 2.0
77 stars 34 forks source link

Feature/add kusto streaming feature #364

Open ag-ramachandran opened 7 months ago

ag-ramachandran commented 7 months ago

Pull Request Description

This is the PR that was for Kusto Streaming. Cherry-picked from the PR #301 and rebased with master

This adds a new option called Stream, that streams data using ManagedStreamingIngest client to Kusto.

Did a couple of large runs to Kusto . While it seems to run satisfactorily, many times it falls back to queued ingest in case the volumes are high

image

Lower volumes seem to be using streaming quite effectively

image

Future Release Comment

[Add description of your change, to include in the next release] [Delete any or all irrelevant sections, e.g. if your change does not warrant a release comment at all]

Breaking Changes:

Features:

Fixes:

github-actions[bot] commented 6 months ago

Test Results

79 tests  +3   79 :white_check_mark: +3   2m 37s :stopwatch: +6s 22 suites ±0    0 :zzz: ±0  22 files   ±0    0 :x: ±0 

Results for commit 7c9a93b5. ± Comparison against base commit 2b70b398.

:recycle: This comment has been updated with latest results.