search

microsoft / sql-server-language-extensions

language-extensions-sqlserver
Other
106 stars 42 forks source link

Streaming Support #32

Closed moranms closed 2 years ago

moranms commented 2 years ago

Are there any plans to incorporate streaming support for language extensions? For my specific use case it would be beneficial for instances where a large volume of data is generated on the scripting side for output. Currently it appears that the Execute interface method is only/can only be called a single time by the Extensibility Framework?

arunguru-msft commented 2 years ago

Streaming support in sp_execute_external_script using r_rowsPerRead is supported for language extensions.

Another option that can be applicable in some cases is partitioning support is also supported for language extensions.

moranms commented 2 years ago

Thanks - I had seen the r_rowsPerRead feature but figured it was only for R and Python baked in via Machine Learning Services, not part of the Language Extensions framework. I don't see anything in the current Extensibility Framework API that would allow one to stream back output results for a custom language extension.

My use case might be a little unorthodox, but I have a C# query that produces a significant amount of data to be output back to SQL. Currently w/o any means of streaming back results, the ExtHost process grows in memory as it collects the script results until the script is complete, at which point it pipes the data back to SQL and ExtHost shuts down. It would be beneficial if the data (the output DataFrame) generated by the C# script could be streamed back as results from the script become available. At the very least it would make the call to sp_execute_external_script more responsive instead of building up a massive collection of data before any results are available. I.e., buffering the entire output result set in ExtHost before sending any results is less than ideal.