Ingestion Time not updated for historical insert

[Sdelausnay]

Hi,

I'm running Light ingest to import some historical data. It is structured as follows

container/telemetry/yyyy/MM/dd.parquet

These files are from 2021.

I have used Light Ingest with following parameter

-creationTimePattern:"telemetry/'yyyy/MM/dd'.parquet"

When I run this command it fails because the soft delete was only set to 30 days on the table. So seems to be working.

logs

``` Failed to ingest 'https://saslaudemo.blob.core.windows.net/history/telemetry/2021/06/12.parquet', Id '6585de5d-3d5d-4002-b928-b7d349687f10'. Operation status is 'Failed'. Failure details: Specified creation time is earlier than table's soft delete period Failed to ingest 'https://saslaudemo.blob.core.windows.net/history/telemetry/2021/06/16.parquet', Id '5aeb52d0-f313-4c95-8048-395e9b78a2b8'. Operation status is 'Failed'. ```

But when I extend the soft delete on the table, the Light Ingest runs fine without any issue. But when I go and look into the ADX table, the ingestion_time() is still the current timestamp of executing the Light Ingest tool and not from the folder of the data.

Is this the expected behavior or am I missing something? Is it another meta data field which is update in the table? Is there a way to evaluate if the data was inserted correctly keeping track of the history so so it follows the soft delete policy?

Thanks in advance.

[ohadbitt]

Hi @Sdelausnay This is expected ingestionTime() is when the data was ingested To see creationTime, run .show table extents - you will see MaxCreatedOn set to the time pattern

[Sdelausnay]

Thank for the clarification!