Azure / azure-kusto-dotnet

Azure Data Explorer (Kusto) SDK for .NET
MIT License
6 stars 3 forks source link

DropByTags and other tags seem to be ignored during ingestion #21

Closed NoTuxNoBux closed 10 months ago

NoTuxNoBux commented 10 months ago

I'm using a IKustoIngestClient created using KustoIngestFactory.CreateStreamingIngestClient in a somewhat standard fashion (see below), but no matter what DropByTags, AdditionalTags, or IngestByTags I specify here, they never make it to the extents created by ingesting the data:

return kustoIngestClient.IngestFromStreamAsync(
    stream,
    new KustoIngestionProperties()
    {
        DatabaseName = kustoConnectionDetails.DefaultDatabaseName,
        TableName = options.TableName,
        Format = DataSourceFormat.json,
        IgnoreFirstRecord = false,
        IgnoreLastRecordIfInvalid = true,
        // This tag is never applied?
        DropByTags = new string [] { "my-tag" },
        ValidationPolicy = new ValidationPolicy()
        {
            IsDetailedErrorReportingEnabled = true,
            ValidationImplications = ValidationImplications.Fail,
        },
        IngestionMapping = new IngestionMapping()
        {
            IngestionMappingKind = IngestionMappingKind.Json,
            IngestionMappingReference = options.MappingName,
        },
    },
    new StreamSourceOptions()
    {
        Compress = true,
        LeaveOpen = true,
    }
);

However, if I use the ICslAdminProvider and try to push similar records using .append TABLE with(tags='[{"drop-by:my-tag"}]') <| some-query, adding the tag works fine.

Since the source code is not available (#20), I can't look into the source to see what I'm doing wrong or what is going on here, exactly. Am I doing something wrong or is this perhaps a bug?

I also tried using the non-ingest library ingestion class, but it doesn't seem to support specifying tags at all (or I'm missing it).

yogilad commented 10 months ago

Hi @NoTuxNoBux , This is a documented limitation of the streaming protocol. https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Ccsharp#limitations