petervandivier / PsAdxArchiver

Generic exporter from Azure Data Explorer to an external table in Azure blob storage.
MIT License
1 stars 0 forks source link

`New-XbBatchBounds` - exponential dropoff #12

Open petervandivier opened 1 year ago

petervandivier commented 1 year ago

New-XbBatchBounds starts to really slow down around 20,000 batches or so. This is about 2 years worth of 1-hour spans. Tune if possible and warn if not.

From a spot-check: batches seconds additional_time rps
8,760 6.6 1,327.3
17,520 40.7 34.1 430.5
26,304 107.4 66.7 244.9
35,064 187.8 80.4 186.7
43,824 332.4 144.6 131.8
52,584 490.2 157.8 107.3
61,368 563.4 73.2 108.9
70,128 940.2 376.8 74.6

runtime batches per second marginal increase

$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1971-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1972-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1973-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1974-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1975-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1976-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1977-01-01 00:00Z' -Step '01:00:00'
$batches = New-XbBatchBounds -Start '1970-01-01 00:00Z' -End '1978-01-01 00:00Z' -Step '01:00:00'
$batches.Count