search

Azure / Kusto-Lightingest

Kusto Lightingest tool
MIT License
2 stars 3 forks source link

LightIngest Will Run Until The Computer Runs Out Of Memory #6

Closed TheTman7 closed 6 months ago

TheTman7 commented 7 months ago

Description: During a LightIngest call with over 3.3 million parquet files and 800gb of compressed data, the LightIngest utility's memory continuously grew. This only appears to happen when items are being discovered. Once all of the items have been discovered and items are being filtered and ingested, the memory stays the same.

Reproduction:

  1. Store millions of parquet files in an Azure storage account (the parquet files I stored were under 1mb compressed).
  2. Ingest the files into ADX.
  3. Note the memory increasing the entire time.

Expected: As a user, I would expect the memory to be capped, have an option for a cap, or for it to release some memory at some point instead of going up the entire time.

Note: Millions of files may seem unnecessary, but when you have data in ADX that goes back to before the 2000's, there can be a lot of files from the continuous export that may need to be ingested at some point (either as a backup or somewhere else).

ohadbitt commented 7 months ago

Hi Thank you for opening this issue - i'm working on some solution

ohadbitt commented 6 months ago

Hi version 12.1.0 is released please try https://www.nuget.org/packages/Microsoft.Azure.Kusto.LightIngest

TheTman7 commented 6 months ago

The issue appears to be resolved! Thanks!