Sharing collections between Kusto kernels results in a System.InvalidOperationException

pshelton-skype commented 2 years ago

Is your feature request related to a problem? Please describe. In Azure Data Explorer, I might craft a KQL query that relies on a collection of objects output by a previous query, such as the following:

let waterspoutStates = StormEvents
  | where EventType == "Waterspout"
  | distinct State;
StormEvents
  | where State in (waterspoutStates)
  | summarize count() by EventType, State

I might want to accomplish the same thing in notebooks using two separate KQL kernels and a C# kernel that processes the content in-between, or to have the output of a parent and child query display in the notebook at the same time, for example:

Sharing a scalar variable from C# to KQL kernels works fine, but attempting to share a collection variable in this fashion results in the following System.InvalidOperationException:

Error: System.InvalidOperationException: Error sharing value 'WaterspoutEvents' from kernel 'kql-help' into kernel 'kql-help'. Cannot support value of Type System.Collections.Generic.List`1[Microsoft.DotNet.Interactive.Formatting.TabularData.TabularDataResource]. There is no CSL type that corresponds to 'System.Collections.Generic.List`1[[Microsoft.DotNet.Interactive.Formatting.TabularData.TabularDataResource, Microsoft.DotNet.Interactive.Formatting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]'.
 ---> System.ArgumentException: Cannot support value of Type System.Collections.Generic.List`1[Microsoft.DotNet.Interactive.Formatting.TabularData.TabularDataResource]. There is no CSL type that corresponds to 'System.Collections.Generic.List`1[[Microsoft.DotNet.Interactive.Formatting.TabularData.TabularDataResource, Microsoft.DotNet.Interactive.Formatting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]'.
   at Microsoft.DotNet.Interactive.SqlServer.ToolsServiceKernel.SetValueAsync(String name, Object value, Type declaredType)
   at Microsoft.DotNet.Interactive.KernelExtensions.ShareValue(Kernel fromKernel, Kernel toKernel, String valueName) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelExtensions.cs:line 236
   --- End of inner exception stack trace ---
   at Microsoft.DotNet.Interactive.KernelExtensions.ShareValue(Kernel fromKernel, Kernel toKernel, String valueName) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelExtensions.cs:line 240
   at Microsoft.DotNet.Interactive.KernelExtensions.<>c__DisplayClass6_0`1.<<UseValueSharing>b__2>d.MoveNext() in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelExtensions.cs:line 213
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass21_0.<<UseMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.InvocationPipeline.InvokeAsync(IConsole console)
   at System.CommandLine.Parsing.ParseResultExtensions.InvokeAsync(ParseResult parseResult, IConsole console)
   at Microsoft.DotNet.Interactive.Commands.DirectiveCommand.InvokeAsync(KernelInvocationContext context) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\Commands\DirectiveCommand.cs:line 34
   at Microsoft.DotNet.Interactive.Kernel.HandleAsync(KernelCommand command, KernelInvocationContext context) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\Kernel.cs:line 257
   at Microsoft.DotNet.Interactive.KernelCommandPipeline.<BuildPipeline>b__6_0(KernelCommand command, KernelInvocationContext context, KernelPipelineContinuation _) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 57
   at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_1.<<BuildPipeline>b__3>d.MoveNext() in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 73
--- End of stack trace from previous location ---
   at Microsoft.DotNet.Interactive.CompositeKernel.LoadExtensions(KernelCommand command, KernelInvocationContext context, KernelPipelineContinuation next) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\CompositeKernel.cs:line 144
   at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_0.<<BuildPipeline>g__Combine|2>d.MoveNext() in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 74
--- End of stack trace from previous location ---
   at Microsoft.DotNet.Interactive.KernelCommandPipeline.SendAsync(KernelCommand command, KernelInvocationContext context) in D:\a\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 48

Rename Waterspouts.txt to Waterspouts.ipynb to demonstrate the repro.

Describe the solution you'd like Sharing collection types such as Lists from a C# or KQL kernel into a KQL kernel should be supported.

Describe alternatives you've considered N/A

pshelton-skype commented 2 years ago

Hi, team. Any updates on this? This would be a blocker for our team's adoption of .NET Interactive Notebooks.

JustinMDotNet commented 2 years ago

I dug into the documentation for Kusto. It looks like we need to use a dynamic CSL type for Kusto rather than a list. Attaching scalar data types for Kusto below.

https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/

pshelton-skype commented 2 years ago

Hi, everybody. Any updates?