Python Interactive with Fable

@dbrattli After trying it for a while, I'm less convinced of the region compilation feature. This is because it forces users to continuously switch between the F# and Python files and maybe not everybody is comfortable with that. There also frequent conflicts when both user and Fable try to write the Python file.

However I'd still like to focus on using Python interactive with VS Code because the Jupyter notebooks are not a good programming environment for F# (unless they can ben open with VS code and get F# support? I haven't tried). So I was wondering if we could use a technique similar to the F# template highlighting extension. That is, we use Fable as a service and create a virtual document in memory with the generated Python code. Then when we capture the "Execute cell" commands and we redirect them (we'll need line mapping between the F# source and the generated Python). This technique could also allow us to integrate Python code into the F# script and provide completion support. If we do the line mapping right maybe we can even get the Python code to understand the F# code thanks to the type annotations.

What do you think? The main drawback is this is more work and requires to maintain a separate VS Code extension. Another option is to collaborate with the experiments @PawelStadnicki is doing with .NET interactive and Fable. I think the experiments are not public yet so I'm not sure how things work, but maybe @PawelStadnicki can provide some insights?

Hi @dbrattli & @alfonsogarciacaro

My last experiments are a part of many proofs of concepts I'm doing in order to prepare a presentation about F# and .NET Interactive. It will take place on 21st June and be co-hosted with .NET Interactive/Try .NET team member Diego Colombo.

Note that this work is focused on some geospatial application rather than any structured library-oriented toward developers. I'm just checking what is possible, leaving to create the proper libraries for later.

But maybe "this later" has just come? :).

By 21st June I have to definitely publish those experiments in one form or another. I'm really keen to cooperate to support it via Fable umbrella. For the sake of following explanations, I have just published a very simple, almost naked F# extension that enables Python (and turfjs library) in the context of JS cell. It is developed via plain Visual Studio, I will add instructions on how to run (produce the expected package) later today. If you have any questions let me know,

But before explaining what the code does let me explain my rationale , as it may be relevant here.

Background:

For the past ~2 years I have had an idea for an application (it doesn't matter here what it exactly does) that should gain F# attention from the mere mortals, as a first language to learn. Taking more traction from C# developers could be still a nice side effect.

I have been starting it all over again since then, mainly because not all the things I needed were technically possible at a certain point in time.

Additionally, .NET developers perceive notebooks as a "toy tool" not worth learning. IMHO, Notebooks, especially .NET Interactive, enforce imagination in order to create something cool and I'm going to take care of it in June :).

But in the last month events are unfolding pretty quickly:

I have been invited to cooperate on the presentation
I discovered cool VS Extensions that do a lot for me for free (in regards to the domain of my application)
.NET Interactive team is about to fix a long-waited bug that was preventing doing pretty amazing things that can be interesting to F# community (basically oriented on JS -> F# code exchange, F# -> JS is working ok)
pyscript.net has been released, it uses pyodide (wasm), which in turn gives us python interoperability almost for free.
pyscript has a slogan "to bring programming for the 99%.". It may sound like a typical startupish "not-possible to accomplish phrase", but at the same time, it gives some direction that "some sort of web-based, interactive programming" is going to gain a lot of traction from the very now. 2k stars daily is unbelievable. Although it is purely a Python community effect,t it will pave the way for others.

Fable.Python

Let's look at Fable.Python in the Python (Jupyter) notebook context. If I'm not mistaken we write F# code that transpiles to Python in the background and has access to full Python (because Python code runs in the Python context). This approach targets mainly Python developers that want to use F# as "a batter Python". To some extent, it also targets F# developers that want to use Python libraries via F# code.

So we have access to Python lang and libs but have to write F# support on our own. Fable.Python gives F# but not autocompletion and no F# libs (not counting bindings)

There is a second audience: F# (.NET) developers that want to use Python sorely because of the libraries.

But in contradiction, they want to run the code in the full .NET context. Although there is SciSharp umbrella of projects that try to mimic it, it still is "not a Python itself".

They most likely want to use Python as an addition to F# or JS side by side and mix this triplet: sky is the limit.

So In .NET Interactive notebooks we have full F# and .NET, with autocompletion. We haven't had Python until Pyodide arrived (wasm).

Besides that .NET Interactive has a lot of other benefits (mind they can also be in the Python ecosystem, but again available from the Python context. I know there are widgets that Dag also addressed):

.NET Interactive can display results as any MIME, including HTML so as a result of F#/Python code we can display full web application created also with F#/JS/Python (via Feliz, Fable.Lit etc) . I'm presenting exactly this on my Twitter recently.
.NET Interactive currently is used mostly via VS Code extension that works also in GitHub Codespaces (gitpod probably too). It is designed for any host, like MS Teams chat for example. But it all means we have JS in the background which is cool because this is the main thing that Fables does :).

.NET Interactive implementation

.NET Interactive is amazing in regards to extensions. Ultimately I can see Fable as a separate kernel in .NET Interactive, mock here:

It is possible to be achieved right now but requires custom .NET Interactive deployment and sharing. It must be a burden to do it right now. I have been informed several months ago that they work on the possibility to add custom kernels very easily to existing instances via configuration (not sure when it will be there thought).

It definitely should be our target as only via proper Kernel we can do anything and hide much more behind the scenes than with second option.

The second option - the one I'm currently using - is to write (F#) Kernel Extension.

The main advantage is that you can share it just by loading a nuget package: )

On loading the F# kernel extension we can run under the cover any code in any available language. For example here is how I load turfjs and pyodide :

For the purpose of my PoCs it is just a raw js code that uses requirejs (the library used internally by .NET Interactive) but I was also able to embed full webpack dist in other scenarios (for pyodide I haven't spent enough time to play with webpack but it should not be that hard).

Now I have turfjs and python function for my purposes in JS cell:

Note that I also can enable it in F#/C# cells but till the JS -> F# bug is not solved, I have only an awful workaround that makes the things too blurry.

In order to use Python libraries, we have to load them via js but for my goal, I created special "magic commands":

This raw approach is quite interesting but we would like to use F# in favor of raw Python. So I created another magic command that copies cell content to .fsx file and runs fable-py under the cover.

Here goes the implementation. I'm storing code in files and running fable-py via CliWrap, not knowing yet how to achieve it in memory:

I'm using the same methods to mix quite advanced stuff like displaying full web apps created with Fable.Lit/Feliz based on F# interactive code written earlier in the notebook.

@alfonsogarciacaro I'm trying to understand this. Is the idea is to embed F# inside Python code? E.g as tripple quoted text and have it rendered as F# in the vscode? An then enable interop between the languages. Almost like an F# type provider for Python?

@alfonsogarciacaro I think this could be very interesting for better interop. But it would also be great if this could also work without vscode and even in a production environment. E.g if we wrapped the code as a doc-string within a class then an @fsharp decorator could even compile the code into a separate backing file, and re-import it dynamically back into the Python code. A quick test (20 lines of code for the decorator).

Screen Recording 2022-05-14 at 22 44 14

What do you think? Something like this combined with F# template highlighting ...

The gif scrolled a bit too quickly, but this is the POC code. It would be a lot more optimized e.g not compiling if the doc-string has not been changed etc:


import importlib
import os
import re
from typing import Any, Type, TypeVar

_A = TypeVar("_A")

def snake_case(name: str) -> str:
    return re.sub("(?!^)([A-Z]+)", r"_\1", name).lower()

def fsharp(cls: Type[_A]) -> Type[_A]:
    lines = getattr(cls, "__doc__", "").splitlines()
    code = os.linesep.join(
        line[4:].strip() if len(line) > 4 else line.strip() for line in lines
    )

    if not os.path.exists("fable"):
        os.mkdir("fable")
    name = f"{cls.__name__}.fsx"
    with open(f"fable/{name}", "w") as f:
        f.write(code)

    os.system(f"dotnet fable-py fable/{name}")

    modname = snake_case(name).replace(".fsx", "")
    module = importlib.import_module(f"fable.{modname}")

    class Wrap(cls):  # type: ignore
        def __getattribute__(self, name: str) -> Any:
            return getattr(module, name)

    return Wrap

#  %%[fsharp]
@fsharp
class MyCode:
    """
    let x = 42

    let add(a, b) = a + b
    """

   # You can also add type hints below: 

    x: int

    def add(self, a: int, b: int) -> int:
        ...

fs = MyCode()

print(f"From F#: {fs.add(10, 30)}")```

Thanks a lot for all the detailed explanation and sorry for the late reply @PawelStadnicki! 🙌 I don't have much experience with .NET Interactive so I needed some time to digest everything :) Really looking forward for your presentation in June!

Looks like thanks to .NET Interactive and your work, .NET and F# developers will already have access to a rich exploratory environment, so maybe compiling F# to Python doesn't bring much more to the table in that case 😅 Although, is it my understanding correct that cross-language interop in .NET interop is mostly done by variable sharing? In that case, F# > Python could still add value, as it allows you to easily call F# functions directly from Python and using Python native data types.

What I'm trying to achieve is an easy (both for users and for us to implement) way so when Fable Snake Island alpha is released (hopefully in a couple of weeks) devs can quickly start playing with F# and Python. In the experience with Fable JS, bindings to libraries have always been the trickiest part to get right, so being able to write F# for data crunching in memory and Python to interact with the libraries would be ideal.

I think @dbrattli proposal meets the requirements 😸 We only need to bring the Fable server back from Fable 2 so we can run Fable as a service instead of booting up each time. And providing a VS code extension similar to https://github.com/alfonsogarciacaro/vscode-template-fsharp-highlight so we can provide completion support for F# inside Python (hopefully it's not too difficult to adapt the extension).

@dbrattli Great example!

It is very interesting which direction would have more usability: running F# in the Python context or running Python in the F# context.

Obviously, both can gain traction. Dag is definitely coming from the first background, it is an enormous community, and letting them learn F# is clearly only a good thing.

On the other hand, I (currently) have focus on F# developers who want to use Python in the full .NET interactive loop. Like calling Python that just invokes some math or data science library and uses the value further in the F# loop. It can be either raw Python via interpolated string or Fable.Python:

the following is compiled to Python first and then executed. It will be more clear when some binding is in place. Do you have some simple example with binding using numpy or math ?

fable-compiler / Fable