Notes on how advanced interactive charting/editing/.... tooling is implemented

[dsyme]

I've been chatting to @Krzysztof-Cieslak about how he and @tpetricek authored the impressive-but-now-defunct F# scripting/charting prototype integrated into Atom. This is also related to FsLab and also iFSharp Jupyter notebooks.

The experience of these is slightly different. First, they all support something in common: they have crucial "HTML printers" for objects. This means when a value is displayed, HTML can be used rather than Text outputs. That's why they look so amazing and are so extensible. AN example of AddHtmlPrinter is in the blog post.

However, the interaction and evaluation model for F# scripting is slightly different in each of the above.

1. In traditional F# REPL scripting, send-to-interactive sends to a process which does incremental evaluation of the results and spews results to stdout, respecting AddPrinter printers based on object type.

   Pro: incremental evaluation without reload of data Con: no extensible visual outputs Con: no true "integration" into host editor, basically just a terminal window, perhaps with editable input Con: no rich "document" is produced by running the script

2. In FsLab+Ionide prototype, the user works in an F# script and Alt-Enter execution of fragments of code always displays the results as HTML in the editor browser panel if an AddHtmlPrinter is available.

   In this setting, the use of fsi.exe is replaced by the use of a bespoke interactive script processor built using FSharp.Compiler.Service. Ionide communicates with this process using a bespoke HTTP protocol on a localhost webserver. This webserver also serves image content.

   Pro: incremental evaluation without reload of data Pro: extensible visual outputs, true "integration" into host editor Con: the script is the "document", there is no overall visual document produced, though you could run another tool to do that Con: needs bespoke F# script processor

3. In FsLab journals today, the user works in an F# script and (if F5 is pressed) a background process watches for saved file changes and evaluates the whole script, generating HTML (or PDF) displayed in a conventional browser. The script processor respects AddHtmlPrinter.

   This approach works in both VS and VSCode, which is why @tpetricek used it for FsLab. For example, in VS or VSCode the user's script can also be evaluated by Send-to-interactive to traditional F# Interactive, but no HTML output is generated (since traditional F# Interactive doesn't know about AddHtmlPrinter) unless the AddPrinter knows to create a browser window for an output.

   The background script processing is today done by a background tool called the fsx2html tool using somewhat stuff from FSharp.Formatting, using old FSharp.Compiler.Service.dll. It's important to note that the fsx2html.exe includes starting a localhost webserver to serve image content - the tool that is processing scripts can serve image content.

   Pro: extensible visual outputs, true "integration" into host editor Pro: no additions to host editor Pro: document-in, document out - you see the output document immediately and focus on it Con: re-evaluates entire script repeatedly, reloading data

4. In iFSharp Jupyter notebooks, something similar to (2) happens, except the processing is done in a Jupyter kernel and results sent back via the Jupyter protocols to populate the cell results with HTML.

   The iFSharp kernel acts as a script processing engine and utilises a fairly recent FSharp.Compiler.Service to do that. It also responds to language service requests to allow checking and other language service features supported by the Jupyter front-end.

   I'm not quite sure if/how iFSharp serves image content - this may be part of the Jupyter kernel protocol, @cgravill may enlighten or we could check the code.

   Pro: extensible visual outputs, true "integration" into Jupyter Pro: document-focused - no separate document window, the evaluation happens in the document itself Pro: incremental evaluation without reload of data Con: iFSharp kernel includes bespoke F# script processor Con: uses .ipynb format to store files, these are not standard in the F# ecosystem

I'll update this with further notes as time goes on. One thing I notice in common is that in each case where HTML outputs are supported an alternative script processing engine is used, rather than F# Interactive as you get it with the Visual F# Tools.

[Krzysztof-Cieslak]

As a note before moving further, I'm a huge fan of what we've done in Atom, but I'm not huge scripts user so I may not be the best person to judge which of those is best.

My few random comments:

• It seems (and @tpetricek can confirm that) FsLab+Ionide and FsLab+fsx2html are using same API for the HTML printers which means they're compatible - you create single HtmlPrinter for both tools.
• I think we could mix what we're doing for Atom with fsx2html to solve the problem of "no overall visual document"
• I believe the current version of FsLab+fsx2html could be done better - using WebSockets to provide auto-refreshing capabilities instead of a user manually needing to refresh the webpage, maybe hosting the page in editor panel instead of the external browser window so the experience seems even more integrated
• I believe that document-focused part of the Jupyter may be also its problem, especially when people try to create bigger/more complex scripts.
• It's also important to notice that notebooks are cell-based, which has it's own pros and cons (@cgravill and @tpetricek will have more thoughts on that, I believe)
• While notebooks have some editor features it's not as good as "real" fully-featured IDE like VS/Ionide

[dsyme]

I guess for completeness we should list out other approaches to live coding experiences for F#, e.g.

• LiveUpdate in Fabulous. Basically replaces some of the logic of a running app, with no state migration
• Live coding in Fable REPL. Basically live launch of JS code for app. No migration of data or app state
• FCSWatch is a live coding tool for F# projects (and since 0.6 includes the ability to send compiled outputs to a http hook, e.g. an app on device or local)
• Bolero HotReload is like Fabuous/Fable hot reload (has been running into some technical issues)

There are many others, please add links below and I'll add them in for completeness.

[smoothdeveloper]

Just adding a note that this is an area of interest for every party that would like to integrate FSI in their product.

[kevmal]

Maybe too old to mention but another approach is what VSLab was doing. Similar to MATLAB, Spyder, Jupyterlab. Separate script editor/ output console/ graphical output. https://archive.codeplex.com/?p=vslab

Jupyter kernels can output multiple mimetype->data pairs and the frontend chooses how to display it: https://jupyter-client.readthedocs.io/en/stable/messaging.html#display-data (My experience is with interacting with kernels using client API as oppose to implementing them).

Biggest issue for me with Jupyter is @Krzysztof-Cieslak comment "While notebooks have some editor features it's not as good as "real" fully-featured IDE like VS/Ionide". Otherwise I quite like the cell based workflow even though coming from earlier notebook UIs like MuPAD/Maple, I felt the IPython/Jupyter notebook experience never quite measured up.

[cgravill]

For 4. with F# in Jupyter notebooks then there’s a few variations. As @kevmal indicates, we set a pairing of mimetype and data, e.g. FSharp.Charting

https://github.com/fsprojects/IfSharp/blob/2776d3693d63e4782b895f80c14ffdb18ff34fcb/src/IfSharp/FSharp.Charting.fsx#L69-L70 Make the data: https://github.com/fsprojects/IfSharp/blob/2776d3693d63e4782b895f80c14ffdb18ff34fcb/src/IfSharp/FSharp.Charting.fsx#L54-L58

For XPlot/Plotly we do something equivalent but instead we send JavaScript that’s capable of drawing the chart: https://github.com/fsprojects/IfSharp/blob/2776d3693d63e4782b895f80c14ffdb18ff34fcb/src/IfSharp/XPlot.Plotly.fsx#L10-L11

Note there are additional methods that draw to png and svg but that’s still done in JavaScript rather than on the server: https://github.com/fsprojects/IfSharp/blob/2776d3693d63e4782b895f80c14ffdb18ff34fcb/src/IfSharp/XPlot.Plotly.fsx#L39-L47 In both those cases we’re mostly relying on Jupyter to handle the UI aspects.

There’s also another variation implemented by @dgrechka which instead of a one-shot send is able to stream over results to their respective charts: https://github.com/fsprojects/IfSharp/blob/2776d3693d63e4782b895f80c14ffdb18ff34fcb/src/IfSharp/Angara.Charting.Dynamic.fsx#L92-L100 (video: https://www.youtube.com/watch?v=DZjZ4tYjoG8)

Given we have these partial bits of "F# output" on the frontend of Jupyter I’ve also been considering pushing more over e.g. Bolero/Fable since that would allow notebook authors to build their visualisations in F# rather than inlining lumps of JavaScript.

I agree about the editor in Jupyter, there have been a lot of efforts towards getting Monaco working but they don’t seem to have taken hold. For Python folks there’s a working Visual Studio Code approach here of connecting to Jupyter: https://code.visualstudio.com/docs/python/jupyter-support

[mathias-brandewinder]

Some quick thoughts on the discussion:

My primary F# data science environment at the moment is VSCode+Ionide. The biggest limitation is charting: I use XPlot/GoogleCharts or Plotly, and constantly have to switch back and forth between VSCode and the browser, which is disruptive. Having rich html outputs in the same spot as the FSI evaluations - which is what the Atom integration did, I believe - would go a long way.

I see Notebooks as serving a complementary purpose. I see their value in sharing exploratory work with others, with input/output all in one document. I personally don't see myself using them for my general work: I like that scripts are "just code", it's not unusual for me to take a script and promote it to a library. I also have a hard time understanding the evaluation model of notebooks, and what is refreshed when a cell is modified and re-run, but that might be just me :)

As a bonus, being able to inspect what is currently in memory in the FSI session would be nice - perhaps something along the lines of what FsEye did. In the end, what I probably want is something close to RStudio (https://www.rstudio.com/products/RStudio/), just in VSCode, for F#.

[dsyme]

@mathias-brandewinder Interesting thanks.

Re notebooks - one possibility is to make F# tooling understand ipynb as a document format. This would make it trivial to use notebooks as libraries - a very important transition step.

That is you could:

1. load "aaa.ipynb" from foo.ipynb, or

2. load "aaa.ipynb" from foo.fsx, or

3. load "bbb.fsx" from a foo.fsx, or

4. load "bbb.fsx" from a foo.ipynb

In each case the ipynb would be accepted via the obvious interpretation of all the top-to-bottom code as a single script, equivalent to an extract-script operation.

There are some subtleties here though - because ipynb are cell-by-cell, they tend to be executed cell-by-cell and earlier executions can perform steps which affect the type-checking of later cells. This is similar to the hacky use of ";;" in .fsx scripts to form a kind of cell. When you #load on such an .fsx file today the F# tooling treats the entire workbook as one thing. I guess the same would apply to #load on a ipynb.

[musheddev]

Flowing from @Krzysztof-Cieslak idea, I had the idea to build a type provider to take a fable charting (d3) script to create a chart model that could be updated and push model changes through a websocket. type LineChart = Chart<"linechart.fsx"> //initializes a shared graphics server let chart = LineChart("chart1") //adds blank chart to internal graphics server //do data science let (chartmodel : LineChart.Model) = //produce a chart model chart.Update(chartmodel) //updates chart through websocket

The main idea being to decouple what a chart looks like from the data science code. The reason I came up with this idea is because I needed to build some custom charts for nonlinear equation graphs.

[cartermp]

Also tagging @jonsequitur

[cgravill]

@dsyme / @mathias-brandewinder it might be worth checking out what the Python folks have done in VSCode for Jupyter notebooks

https://code.visualstudio.com/docs/python/jupyter-support

When you open a ipynb it offers to extract the script:

Then you can run a cell at a time:

That gives much more of a sequential flow, rather than the Jupyter UI which allows you to run cells out of their document order.

This isn't implemented with F# notebooks at present.

[granicz]

For an another advanced use case, here is what CloudSharper (implemented with WebSharper) had around late 2013 - click image for a video showing charting static values and rows coming from a database. Subsequent product versions featured a more modern UI but the same functionality, and our goal was to enable hosting the underlying code services on any remote server or a cloud provider (hence the name). Although still available online, the project has been put on the backburner (current users need an old F# setup on their machines and installing a local component to work) and will likely stay there until we replace the entire client-side architecture with a Bolero-based WASM implementation.

You can type code into the Interactive shell directly, as well as send code via Ctrl+Enter from the code tabs. This was implemented via a custom FSI that lit up certain values through the WebSharper compiler (in particular, Elts or pagelets - WebSharper.Html's representation for HTML with events/client-side functionality, and functions marked for client-side use), injecting HTML+JS for each response. For instance, with a single line of code, you could define an HTML button with a click event, and a button would appear. You could use also any WebSharper extension as well (!) and the output would link their underlying JS libraries automatically using WebSharper's dependency tracking (as with the charts in the video). Given that the interactive session was kept alive (with the ability to reset), new code could rely on earlier values, and even call server-side functions (which were hosted in the FSI process) to emulate client-server calls.

It also has a literate programming mode where "workbooks" with live code can be implemented with Run-in-interactive buttons.

[tylerhartwig]

• Bolero HotReload is like Fabuous/Fable hot reload (has been running into some technical issues)

As for the technical issues I was running into, those were utilizing some of the bits of Portacode incorrectly. I was able to work around them, but have since switched my strategy to use FCSWatch as I noticed Portacode seems to be merging with FCSWatch and as Don mentioned to me, there's no need to do interpretation running on mono-wasm.

I am planning on dynamically maintaining state in the reload as well.

I was also unaware that FCSWatch added web hooks recently, that will be very helpful for my development.

[sgoguen]

I am really amazed at all of the incredible work being done here, but I figured I'd throw something I've been doing as a side proof-of-concept project that I've been meaning pick up again. (I apologize if I go long, brevity is not my thing.)

Last year I was playing around with a tool (see it running here) that I forked from @gusty. It loosely resembles the FsLab + Ionide setup in that you're adding a custom printer to the FSI. When you evaluate a line, it formats the values as HTML, sends it to a Suave server, then pushes the HTML to the browser via web sockets. On the client, there's some really dumb JavaScript that renders whatever you send it. There's so much lacking. At some point, I was thinking I'd push up a DOM instead so I could leverage something like React to reconcile the updates. From a UI perspective, you have to open a separate browser. However, once you did that, you could keep that browser open because it will automatically reconnect when you find yourself restarting the web server.

However, when I came across the VSCode Browser Preview extension in January, the first thing that occurred to me was I could now have the VS Code split screen experience. It worked (same tweet as above), but it's definitely rough around the edges. It can render blurry, sometimes there's latency and I needed to push HTML twice for it to take.

I'm still a huge LINQPad fan and lately I started using Elmish in LINQPad. My fork bastardizes Elmish by eschews type safety because it effectively uses LINQPad's object dumper with the Elmish DSL (children can be anything). I don't condone the solution, but it does make creating throwaway prototypes fun. My next step was to make the HTML object dumper pluggable so I could get the Elmish bastard running on my Mac.

My ideal setup would be something that gives me the option to use it like FSI, LINQPad or something that does hot reloading:

• Loading up VS Code on my Mac with .NET Core only (no Mono dependencies)
• Opening an fsx file that has a reference to all my packages
• Hit a hot key (like F5 or whatever) to run it.
• Watch my process update the browser on right split screen.
• Evaluate a line or snippet while the process is running.
• Make a change and hit my hot key to restart my fsi process. (Or use a file watcher to trigger it)

I can see how all the incredible work being done by you all is coming together and getting us so close to realizing this dream. Maybe it's the caffeine talking, but seeing all the progress is exciting!

[colombod]

I am a big supporter of the way LINQPAd gives access to dom manipulation for rendering. I also like growth that Bokeh is going through.

The approach is interesting as it provides bidirectional binding, such feature can be used to interactive drill down on histogram bins for example (data zooming, or LOD more than graph zooming).

Since there are bindings already for python and Jupyter interaction I suppose that a decent .net bindings for it would open all .NET ecosystem to a wide set of new application for chart and advanced exploratory data analysis

[jdh30]

Fascinating technical discussion.

Anyway, I'd add something about business and applications. From my point of view there are three relevant groups of people who would use this kind of functionality:

1. Developers, e.g. IT, quants and data scientists.
2. Semi-technical content producers, e.g. Traders, actuaries, analysts.
3. Non-technical content consumers, e.g. customers, business people and stakeholders.

They have different requirements:

1. Full interop and access, e.g. raw F# code running on .NET calling out to SQL Server.
2. Ease of use in terms of the language and tooling, e.g. edit some F# script where you have Intellisense and all relevant libraries pre-accessible and click "Save" to have it uploaded to the server where group 3 can see it. No DLLs/binaries/executables/installers/VCS/build systems/fsproj hacking etc.
3. Ease of use in terms of accessibility, i.e. all output in the browser.

Is a web-based IDE for Group 1 feasible or do they need VS/Code? Group 2 could probably use either Windows or web based tooling. Everyone probably wants more than charts. In some cases (e.g. coding tutorials) you might want both code and output charts visible to the end user, e.g. integrate Tomas' FSharp.Formatting tool. In other cases (e.g. CMS, analytics dashboards, reporting) you probably want the output charts embedded in documents.

I suspect it would be best to keep the current ability to develop GUI apps in FSI with shared state for group 1 and add web output of charts but my feeling is that the latter would benefit more from having everything running in the browser (from the editor with Intellisense to the generated output), at least for groups 2 and 3. And I have no idea how the results would be put into documents.

[cartermp]

Closing old discussion. I think a lot of this has worked its way over into improvements here: https://github.com/dotnet/interactive/

I would suggest anyone interested to try things out in VSCode notebooks and file issues on the interactive repo about how to handle things https://github.com/dotnet/interactive/#visual-studio-code