microsoft / delta-kusto

Engine able to compute delta between ADX clusters (and/or Kusto scripts) and generate update scripts
MIT License
46 stars 18 forks source link

Merge policy is always triggered #175

Open matipos2 opened 3 months ago

matipos2 commented 3 months ago

Merge policy is being always triggered for execution by delta-kusto even if there is no change. Maybe the option Lookback.Kind is messing a bit, because sometimes it's being read as text, sometimes as number. For example Lookback.Kind = “All” or 2.

Status on database:

.show table BEO_TagDataProcessed policy merge 

"PolicyName": ExtentsMergePolicy,
"EntityName": [MyDB].[MyTable],
"Policy": {
  "RowCountUpperBoundForMerge": 16000000,
  "OriginalSizeMBUpperBoundForMerge": 30000,
  "MaxExtentsToMerge": 100,
  "MaxRangeInHours": 168,
  "AllowRebuild": true,
  "AllowMerge": true,
  "Lookback": {
    "Kind": "All",
    "CustomPeriod": null
  },
  "Origin": "Admin"
},
"ChildEntities": ,
"EntityType": Table,
"EffectivePolicy": {
  "RowCountUpperBoundForMerge": 16000000,
  "OriginalSizeMBUpperBoundForMerge": 30000,
  "MaxExtentsToMerge": 100,
  "MaxRangeInHours": 168,
  "AllowRebuild": true,
  "AllowMerge": true,
  "Lookback": {
    "Kind": "All",
    "CustomPeriod": null
  },
  "Origin": "Admin"
}

Result by .show database schema as csl script:

"DatabaseSchemaScript": .alter table MyTable policy merge @'{"RowCountUpperBoundForMerge":16000000,"OriginalSizeMBUpperBoundForMerge":30000,"MaxExtentsToMerge":100,"MaxRangeInHours":168,"AllowRebuild":true,"AllowMerge":true,"Lookback":{"Kind":2,"CustomPeriod":null},"Origin":1}'

My script:

.alter table MyTable policy merge
```{
  "RowCountUpperBoundForMerge": 16000000,
  "OriginalSizeMBUpperBoundForMerge": 30000,
  "MaxExtentsToMerge": 100,
  "LoopPeriod": "01:00:00",
  "MaxRangeInHours": 168,
  "AllowRebuild": true,
  "AllowMerge": true,
  "Lookback": {
    "Kind": "All",
    "CustomPeriod": null
  }
}```

I also tried with, but the result is the same:

.alter table BEO_TagDataProcessed policy merge
```{
  "RowCountUpperBoundForMerge": 16000000,
  "OriginalSizeMBUpperBoundForMerge": 30000,
  "MaxExtentsToMerge": 100,
  "LoopPeriod": "01:00:00",
  "MaxRangeInHours": 168,
  "AllowRebuild": true,
  "AllowMerge": true,
  "Lookback": {
    "Kind": 2,
    "CustomPeriod": null
  }
}```