microsoft / delta-kusto

Engine able to compute delta between ADX clusters (and/or Kusto scripts) and generate update scripts
MIT License
47 stars 19 forks source link

Existing ADX function crashes delta-kusto job #141

Closed sflandergan closed 1 year ago

sflandergan commented 1 year ago

I tried to synchronize a schema from a script with an existing database. The database has some tables, materialized views as well as views defined. Unfortunately the latter crash the Job:

Error:  Issue in running job 'push-script-to-dev'
  Error:  Issue parsing script
  Error:  .create-or-alter function with (skipvalidation = "true", view = true) Assets() {
AssetChanges
| where eventType != "assetDelete"
| project-away eventType
}
    Error:  There should be one-and-only-one Property name but there are 0
    Error:  .create-or-alter function with (skipvalidation = "true", view = true) Assets() {
AssetChanges
| where eventType != "assetDelete"
| project-away eventType
}
    Exception encountered:  DeltaKustoLib.DeltaException ; There should be one-and-only-one Property name but there are 0
    Stack trace:     at DeltaKustoLib.CommandModel.CodeHelper.GetUniqueDescendant[TElement](SyntaxElement parent, String descendantNameForExceptionMessage, Func`2 predicate)
   at DeltaKustoLib.CommandModel.CreateFunctionCommand.<>c.<GetProperties>b__23_1(SeparatedElement p)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at DeltaKustoLib.CommandModel.CreateFunctionCommand.GetProperties(FunctionDeclaration functionDeclaration, QuotedText& folder, QuotedText& docString)
   at DeltaKustoLib.CommandModel.CreateFunctionCommand.FromCode(SyntaxElement rootElement)
   at DeltaKustoLib.CommandModel.CommandBase.CreateCommand(String script, KustoCode code, Boolean ignoreUnknownCommands)
   at DeltaKustoLib.CommandModel.CommandBase.ParseAndCreateCommand(String script, Boolean ignoreUnknownCommands)
  Exception encountered:  DeltaKustoLib.DeltaException ; Issue parsing script
  Stack trace:     at DeltaKustoLib.CommandModel.CommandBase.ParseAndCreateCommand(String script, Boolean ignoreUnknownCommands)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()
   at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1)
   at DeltaKustoLib.CommandModel.CommandBase.FromScript(String script, Boolean ignoreUnknownCommands)
   at DeltaKustoIntegration.Kusto.KustoManagementGateway.DeltaKustoIntegration.Kusto.IKustoManagementGateway.ReverseEngineerDatabaseAsync(CancellationToken ct)
   at DeltaKustoIntegration.Database.KustoDatabaseProvider.DeltaKustoIntegration.Database.IDatabaseProvider.RetrieveDatabaseAsync(CancellationToken ct)
   at delta_kusto.DeltaOrchestration.RetrieveDatabaseAsync(IDatabaseProvider currentDbProvider, String db)
   at delta_kusto.DeltaOrchestration.ProcessJobAsync(MainParameterization parameters, IKustoManagementGatewayFactory kustoGatewayFactory, IFileGateway localFileGateway, String jobName, JobParameterization job)
    Exception encountered:  DeltaKustoLib.DeltaException ; There should be one-and-only-one Property name but there are 0
    Stack trace:     at DeltaKustoLib.CommandModel.CodeHelper.GetUniqueDescendant[TElement](SyntaxElement parent, String descendantNameForExceptionMessage, Func`2 predicate)
   at DeltaKustoLib.CommandModel.CreateFunctionCommand.<>c.<GetProperties>b__23_1(SeparatedElement p)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at DeltaKustoLib.CommandModel.CreateFunctionCommand.GetProperties(FunctionDeclaration functionDeclaration, QuotedText& folder, QuotedText& docString)
   at DeltaKustoLib.CommandModel.CreateFunctionCommand.FromCode(SyntaxElement rootElement)
   at DeltaKustoLib.CommandModel.CommandBase.CreateCommand(String script, KustoCode code, Boolean ignoreUnknownCommands)
   at DeltaKustoLib.CommandModel.CommandBase.ParseAndCreateCommand(String script, Boolean ignoreUnknownCommands)

My job definition looks like this:

sendErrorOptIn:  true
failIfDataLoss:  true
jobs:
  push-script-to-dev:
    current:
      adx:
          clusterUri: https://<omitted>.westeurope.kusto.windows.net
          database:  dev
    target:
      scripts:
        - filePath:  other-state.kql
    action:
        filePath:  dev-update-v1.kql
#        pushToCurrent:  true
tokenProvider:
    login:
        tenantId: <omitted>
        clientId: <omitted>
        secret:  <omitted>

This is what .show function Assets returns on the current ADX dev database:

"Name": Assets,
"Parameters": (),
"Body": {
AssetChanges
| where eventType != "assetDelete"
| project-away eventType
},
"Folder": ,
"DocString": 
vplauzon commented 1 year ago

This is the same issue as this one.

It is fixed and will be available in next version.

vplauzon commented 1 year ago

Fixed in 0.14.0.