Python Support for Fable

[dbrattli]

Description

This issue is a discussion about Python support for Fable. A POC has been made to show that this is possible. However, quite a lot of work is needed to:

1. Add decent support for Python
2. Make Fable more JS agnostic and support multiple target languages.

Use cases:

• F# running in Jupyter notebook with Python kernel, i.e similar to what is done with e.g Hy and Calysto Hy.
• F# scripting using Python for reduced friction
• More easily support embedded environments like the micro:bit and Rasberry PI.
• Sharing data model between F# and Python code

Things to discuss and decide:

• Should Python for F# be more Pythonic than .NET? There is already .NET support for Jupyter which we should not try to replace or compete with. Fable for Python should perhaps target Python instead of .NET (if possible) and be a more natural fit for Python developers looking to F#.
• Should we try to be compatible with Babel AST or should we diverge into our own Python AST. This is perhaps inevitable, but Babel gives us a good starting point.
• Should we for Fable (as with Peeble) or should we stay within Fable and have Fable support multiple languages. Having our own repo gives freedom and speed (by not needing to care about JS), but we will risk being left behind and obsoleted over time. My feeling is that we should become part of Fable.
• Python basic types is also different from F#. E.g int has an arbitrary length. Should we emulate .NET (machine) types to be compatible or should we expose the Python int type as an F# int?

Installing the POC

The source code of the POC currently lives here:

• https://github.com/dbrattli/Fable/tree/python (https://github.com/fable-compiler/Fable/pull/2345)

1. Install the latest Python 3.9 from https://www.python.org or brew install python@3.9 on Mac. Note that python may be available on your system as python, python3 or both.
2. Clone both repo and switch to the python branch.
3. Build fable-library for Python: dotnet fsi build.fsx library-py
4. To run tests: dotnet fsi build.fsx test-py
5. In Fable, edit QuickTest.fs to some simple F# code. Then run dotnet fsi build.fsx quicktest-py from the top directory.

Now you can edit both Fable and Expression code and the example code will recompile when needed. Note you need an additional save to QuickTest.fs to trigger recompile after modifying Fable code. The python file will be available as quicktest.py. A nice demo is to view both F#, JS, and Python in vscode at the same time. Then you will see how the code is being transformed when you save the F# file:

You can run the generated Python code in the terminal:

$ python3 quicktest.py

Links

A few relevant links:

• https://github.com/dbrattli/Fable/tree/python
• https://github.com/dbrattli/Fable.Jupyter
• https://github.com/cognitedata/Expression/tree/fable
• https://github.com/thinkbeforecoding/peeble
• https://docs.python.org/3/library/ast.html
• https://github.com/babel/babylon/blob/master/ast/spec.md
• https://fable.io/docs/dotnet/numbers.html

Feel free to contribute with discussions, comments, ideas, and code 😍

[alfonsogarciacaro]

As we talked in Discord @dbrattli, I think you can check the type of the callee to see if it's an anonymous record. But if you need a more systematic approach it should be ok to add more info in FieldGet, though at this point we probably should declare a separate record to minify breaking changes and avoid confusion with other boolean fields (F# should have more strict union case field names btw):

type FieldGetInfo =
    { Name: string
      IsMutable: bool
      IsAnonymousRecord: bool }

type GetKind =
    | FieldGet of info: FieldGetInfo
    | ...

[dbrattli]

Thanks @alfonsogarciacaro. That worked! 🎉

Next issue:

testCase "Map.IsEmpty works" <| fun () ->
    let xs = Map.empty<int, int>
    xs.IsEmpty |> equal true
    let ys = Map [1,1; 2,2]
    ys.IsEmpty |> equal false

For JS this compiles to:

Testing_testCase("Map.isEmpty works", () => {
    Testing_equal(true, isEmpty_1(ofSeq([], {
        Compare: (x_1, y_1) => compare(x_1, y_1),
    })));
    Testing_equal(false, isEmpty_1(ofSeq([[1, 1]], {
        Compare: (x_2, y_2) => comparePrimitives(x_2, y_2),
    })));
})

Note that ofSeq is called with two arguments. However, ofSeq only takes a single argument:

let ofSeq elements =
    Map<_, _>.Create elements

export function ofSeq(elements) {
    return FSharpMap_Create(elements);
}

Why is that object expression with Compare added when ofSeq is called?

[alfonsogarciacaro]

Hmm, I need to look into this. Looks like a bug, ofSeq should accept a comparer (similar functions like MapTree.ofSeq and Set.ofSeq do). The setup is a bit complicated so maybe at some point things went out of sync: We have a file named ReplacementsInject.fs that is used by Replacements to indicate which methods need argument injection. There's a script somewhere to generate this file automatically but it's been a while we haven't used it and I'm not sure if it works with latest FCS version. I will check, thanks for pointing it out!

[dbrattli]

I added a temp fix. Ready for review 🎉 https://github.com/fable-compiler/Fable/pull/2345

[dbrattli]

@alfonsogarciacaro @ncave I'm having a bit of trouble with Array and Array.slice. It looks to be handled by Helper.InstanceCall(ar, "slice", t, [lower; upper], ?loc=r) in Replacements.fs. It currently tries to call obj.slice(), but I need to rewrite it to obj[x:y]. I was hoping it would use my subArrayImpl in Native.fs, but it's not. How does this work?

In Native.fs for Python I have:

[<Emit("$0[$1:$1+$2]")>]
let inline subArrayImpl (array: 'T []) (start: int) (count: int) : 'T [] = nativeOnly

Do we need to have it call a slice function in Array.fs and let that use subArrayImpl in Native.fs? E.g Helper.LibCall(com, "Array", "slice", t, args, i.SignatureArgTypes, ?loc=r) ?

[alfonsogarciacaro]

We probably should have a Replacements module for each language, I'm actually amazed that you made it work for Python as is. I would start by duplicating the module and then you're free to make any modification needed for python. I will send a PR to make this change. The good news is there are only a couple of functions that are called from FSharp2Fable so it shouldn't be hard to abstract them. We actually need to abstract other operations like identifier sanitization, keyword check, etc, so I'll try to make a Compiler interface that can implement different helpers for each language.

As a side node, the Replacements modules is quite complex because I added too much magic at the beginning. Of course it's ok to reuse whatever you can from the existing code, but ideally we should aim at having as many F# modules as possible where redirections can be done automatically as in the latest modules written by @ncave (e.g. StringBuilder) https://github.com/fable-compiler/Fable/blob/c78f584b7622fdd1116e8194ca5bc2dba4c05a13/src/Fable.Transforms/Replacements.fs#L1324-L1336

[dbrattli]

@alfonsogarciacaro That sounds great! What I currently do is to transform the output of Replacements, so it would be great to have a separate Replacements for Python. That should clean up a few things, and perhaps also make it easier to produce cleaner more idiomatic / pythonic Python code. I'll take a look at the StringBuilder code to see of I can understand how things work.

[dbrattli]

Current Status:

test_arithmetic.py .......................                                                                                                                                                                             [  3%]
test_array.py ....................................                                                                                                                                                                     [  9%]
test_async.py .............                                                                                                                                                                                            [ 11%]
test_comparison.py .............................                                                                                                                                                                       [ 15%]
test_custom_operators.py ..........                                                                                                                                                                                    [ 16%]
test_date_time.py ........                                                                                                                                                                                             [ 18%]
test_enum.py ......................                                                                                                                                                                                    [ 21%]
test_enumerable.py ...                                                                                                                                                                                                 [ 22%]
test_fn.py ..                                                                                                                                                                                                          [ 22%]
test_list.py ......................................................................................                                                                                                                    [ 35%]
test_loops.py ...                                                                                                                                                                                                      [ 35%]
test_map.py .......................................                                                                                                                                                                    [ 41%]
test_math.py .                                                                                                                                                                                                         [ 42%]
test_option.py ................................                                                                                                                                                                        [ 47%]
test_py_interop.py ....                                                                                                                                                                                                [ 47%]
test_record_type.py ...........                                                                                                                                                                                        [ 49%]
test_reflection.py .............                                                                                                                                                                                       [ 51%]
test_result.py ......                                                                                                                                                                                                  [ 52%]
test_seq.py ...............................................................................................                                                                                                            [ 66%]
test_seq_expression.py .............                                                                                                                                                                                   [ 68%]
test_set.py ...............................................                                                                                                                                                            [ 75%]
test_string.py ......................................................................................................................                                                                                  [ 94%]
test_sudoku.py .                                                                                                                                                                                                       [ 94%]
test_tail_call.py ................                                                                                                                                                                                     [ 96%]
test_tuple_type.py ........                                                                                                                                                                                            [ 97%]
test_union_type.py ..............                                                                                                                                                                                      [100%]

==================================================================================================== 653 passed in 21.35s ====================================================================================================
Build finished successfully

[dbrattli]

Closing this issue now that Python is supported (as alpha). Please add any Python related issues to Fable.Python