search

microsoft / delta-kusto

Engine able to compute delta between ADX clusters (and/or Kusto scripts) and generate update scripts
MIT License
46 stars 18 forks source link

Unable to use Delta Kusto reliably to maintain ADX tables and mappings #128

Closed delarooster closed 1 year ago

delarooster commented 1 year ago

I'm attempting to utilize this project to help put my ADX tables under source control and am failing miserably. I'm starting from the presupposition of initializing a brand-new database from scatch without any preexisting schema or data.

There is a script file that defines two tables. Full repository here -> Io.Adx.Kusto

  1. Currently my pipeline is set up to look at the existing database and retrieve current state (as a potential rollback later, if required).

  2. I then compare my schema script against the existing database to create a delta.

  3. As the documentation suggests, I publish these files as an artifact to be retrieved by the next step.

  4. In the next stage I retrieve the published artifact and attempt an update of the ADX database, using the published artifact as the source.

A few observations:

  1. Is there a documented best-practice for passing in the desired state of a database via a script file? I had originally included multiple script files (one for each table, I've since reduced the tables to two, removed mappings, policies, to help alleviate questions while debugging) and in my latest iteration put both table .create functions into a single file.
  2. During the push-state job it appears the current and target are flip-flopped parameters: 
    push-state:
    current:
        adx:
            clusterUri: to-be-overriden
            database: to-be-overriden
    target:
        scripts:
            - filePath: ../schema/script.kql

    Throughout the documentation I read that current is the source and target the desired instance to update from current, but here whenever I set it up as current as a file (either my schema or the created delta file), I continually get this error: 

    Error:  Issue with the following parameter override:  'System.String[]'
Exception encountered:  System.Collections.Generic.KeyNotFoundException ; The given key 'DeltaKustoIntegration.Parameterization.AdxSourceParameterization' was not present in the dictionary.
  3. Similar to point 2 above when running delta-dev I use my scripts as the current and target as the adx database I want to initialize, but it appears the program is attempting to compare what's in the ADX db as the source of truth instead of my scripts. 
    delta-dev:
priority: 1
current:
    scripts:
        - filePath: ../schema/script.kql
target:
    adx:
        clusterUri:  to-be-overridden
        database:  to-be-overridden

    Delta output is as follows:

    
//  Drop Tables

.drop table Connectivity

.drop table Telemetry

// Alter Caching Policies

.alter database Dvusahoth policy caching hot = 31d

// Alter Merge Policies

.alter database Dvusahoth policy merge { "RowCountUpperBoundForMerge": 16000000, "OriginalSizeMBUpperBoundForMerge": 30000, "MaxExtentsToMerge": 100, "LoopPeriod": "01:00:00", "MaxRangeInHours": 24, "AllowRebuild": true, "AllowMerge": true, "Lookback": { "Kind": 0, "CustomPeriod": null }, "Origin": 2 }

// Alter Retention Policies

.alter database Dvusahoth policy retention { "SoftDeletePeriod": "365.00:00:00", "Recoverability": "Enabled" }

Appears to be the exact opposite of what I'm looking to create.  And if I attempt to flip (like in [`push-state-to-db-parameters.yaml`](https://github.com/delarooster/Io.Adx.Kusto/blob/sandbox-ext/pipelines/push-state-to-db-parameters.yaml)) I get the same error as point 2.

4. The only thing that *does* currently work is the first time I run a setup on a bare database using my desired schema:

sendErrorOptIn: true jobs: push-state: current: adx: clusterUri: to-be-overriden database: to-be-overriden target: scripts:

Hoping to use this tool, or another, to manage my ADX databases using source control. Big big use case that would be highly beneficial to my team!

Many thanks, Austin

vplauzon commented 1 year ago

After discussion, the mapping error was due to a regression introduced in .show db schema (Kusto command used by DK) that has since been resolved.