MBrace.Sql roadmap and feature support over the coming months

[bruinbrown]

We recently gave a brief demonstration of MBrace.Sql at the F# Exchange 2017. In this demonstration, we showed off some of the basic features of the library which open MBrace as a more accessible platform to as many developers as possible. Our aim with MBrace.Sql was to make data exploration and analysis both easy to get into for those who might have MBrace infrastructure in place already, and to make extensible data analysis easy by providing a means of creating tooling capable of bridging the gap between those who need the ability to fall back to more complex operations whilst still having the simplicity of data analysis using SQL.

In it's current state the MBrace.Sql project is in a very early alpha stage in that it's usable to a limit. The current project allows for performing fairly basic select statements across a range of file types. This is quite obviously a fairly basic feature set, however, there is clearly room for more growth. This issue lists some of the functionality below which could be added to the project and what would be required to implement them in each case along with some degree of effort required as well.

• [ ] Support for scalar UDFs - Easy - This feature would allow the use of UDFs in select statements to map from one value to another. An example of this would be capitalisation of strings
• [ ] Support for Aggregate queries - This requires initial support for group by adding in for which the basic components are already there, after this a standard library of aggregate functions would need developing
• [ ] Support for Aggregate UDFs - Extends the above aggregate queries by allowing custom functions similar to scalar UDFs
• [ ] Joins - Allows joins of datasets. This requires some understanding of the workflow of how joins will be used within MBrace.Sql, i.e. should joins allow joins by file or joins by dataset
• [ ] Memory pressure improvements - The current implementation relies on a Map of a string to a discriminated union which creates a fairly large memory footprint
• [x] Speed improvements - Currently there is experimental support for code generation through quotations in an effort to improve performance, there are a couple of features which are not yet implemented using quotations yet, one of which is case expressions
• [ ] Documentation - The documentation is currently non-existent, however, this is caused largely by the alpha nature of the project
• [ ] Packaging and transfer to the MBrace organisation - The project is currently several scripts and should be packaged together into a DLL shipped via NuGet along with a potential transfer to the MBrace organisation assuming the project fits in with the broader aims

This is clearly a large amount of work to undertake and so it's important for us to understand whether this is something which the broader MBrace community believes could have influence on if it would be a driver or blocker to adoption of MBrace for you. At the F# Exchange, we also listed a number of other potential improvements which could be made to MBrace, including further work on the graph layer on top of CloudFlow, a distributed machine learning library on top of CloudFlow or an IoT scale streaming platform. If the broader community thinks there is more value in one of these libraries instead of focussing on a SQL layer then this will likely influence how much effort is put into the MBrace.Sql project over the coming months

[isaacabraham]

A few other points you might want to add: -

• Type Provider which could both provide compile-time checking of SQL (potentially even against the data source, although that would be a V2 I would imagine) as well as a strong output instead of a simple Map.
• Integration with the SQL Provider AST codebase.

[AviAvni]

I don't use Mbrace at production only playing with it And I need to look more deep on this repo but if someone can help me with my PR of graph processing with mbrace it will be great