[Python] Publish the option --language Python to make it easier to experiment

Zaid-Ajaj commented 2 years ago

Description

I would love to be able to start using --language Python and start experimenting with it. Even though it is not "complete" yet and there are some issues, it will make it easier for users to get a feel of how it works and experiment with what is supported and what isn't.

I propose to publish it as part of Fable and document as it as alpha by logging the following message when compilation starts

Compiling to Python is still in _alpha_ phase and is currently only for experimentation.

I am already getting feature requests to make libraries and tools F#->Python compatible (Femto & Fable.Remoting) so it will be an interesting domain to explore.

cc @dbrattli

P.S. I have seen Fable.Python but I would prefer not having to compile Fable myself if possible 😄

Repro code

N/A

Expected and actual results

Expected to be able to compile a simple console app

mkdir FableTest
cd FableTest
dotnet new console -lang F#
dotnet new tool-manifest
dotnet tool install fable
dotnet fable --language Python
python Program.fs.py

Related information

Fable version: dotnet fable --version 3.4.7
Operating system: Windows 10

Happypig375 commented 2 years ago

I am already getting feature requests to make libraries and tools F#->Python compatible (Femto & Fable.Remoting) so it will be an interesting domain to explore.

Hey, that's me

dbrattli commented 2 years ago

Hey, thanks for the interest. Fable Python is already available as a dotnet tool as fable-py. See Fable.Python for some instructions, e.g:

> dotnet tool install fable-py --version 4.0.0-alpha-005
> dotnet add package Fable.Core.Experimental --version 4.0.0-alpha-006

Then you don't need to set the --language. Note that Fable.Core.Experimental is Fable.Core but with Python sources, so you use it as normal Fable.Core namespace.

See e.g the example project Timeflies on how to setup a Python project.

Note: currently the beyond branch containing Python is not merged with main (either way) but this is something that is being worked on. However, python will probably be available as a separate tool anyways, or at least that's the current thinking.

PS: Fable.Python only contains the type bindings, not the compiler.

dbrattli commented 2 years ago

@Zaid-Ajaj Perhaps we need to do a Twitch session together? 😉

ncave commented 2 years ago

I agree we need to surface the progress in Fable beyond branch in some form, perhaps an alpha repl that has the option to change the generated language, and is auto-updated on merging in beyond.

Zaid-Ajaj commented 2 years ago

@dbrattli Thanks a lot for the instructions! I am trying out on my machine and it is amazing 😍 I will play a little bit more with it and let you know how it goes. Writing bindings is already very familiar:

open System
open Fable.Core

[<Emit "open($0, 'r').read()">]
let readFile (path: string) : string = nativeOnly

[<Import("join", "os.path")>]
let combine(path: string, part: string) : string = nativeOnly

let filePath = combine(__SOURCE_DIRECTORY__, "content.txt")
let contents = readFile filePath 

Console.WriteLine contents

@Zaid-Ajaj Perhaps we need to do a Twitch session together? 😉

Love the idea! I'll try to figure out how to do dual-streaming with OBS

dbrattli commented 2 years ago

@Zaid-Ajaj Great to hear. There's a lot of things left to fix, but feel that we are approaching critical mass with this port. One gotcha to be aware of is that <OutputType>Exe</OutputType> will currently generate absolute imports in Python, while <OutputType>Library</OutputType> (or default) will generate relative imports. This is because a Python program cannot use relative imports, while a library may not know where it's mounted or imported from, so safer to use relative imports in libraries. I know many do not set output type for e.g net5, but otherwise I would need to detect the presence of an entry-point.

Zaid-Ajaj commented 2 years ago

Hi @dbrattli I am having trouble building a library with fable-py

I made the repository Fable.SimpleJson.Python which is a modified version of Fable.SimpleJson without the JS bits but I am unable to run the tests.

git clone https://github.com/Zaid-Ajaj/Fable.SimpleJson.Python.git
cd ./Fable.SimpleJson.Python
dotnet tool restore
cd ./tests
dotnet fable-py 
python program.py

Some problems are as follows:

Paths are weird

when compiling using dotnet fable-py I think it should generate something that works out of the box but here I have to manually edit the imports

# Compiled code: doesn't work
from fable_modulessrc.json_converter import Convert_fromJson
from fable_modulessrc.simple_json import SimpleJson_parseNative
from fable_modulessrc.type_info_converter import create_type_info

see how fable_modules was concatenated with src which doesn't work. The only way I tried to make it run was using --outDir so that the packages have a common parent package but even then I had to manually edit the imports

Using `unint32` on a `float`

// doesn't work
let value = 2.0
let converted = uint32 value

compiles to

value = 2
converted = value >>> 0

Union reflection

The functions make_union and get_union_fields are not implemented :/

@alfonsogarciacaro is it OK to track python related issues here as well?

dbrattli commented 2 years ago

@Zaid-Ajaj I'll have a look. The generated code looks to be wrong so I'll try to fix it. But even then, there are a few things you need to be aware of.

Python imports cannot reference out of their package/directory. Thus you cannot import e.g src from something in test without altering sys.path. So project references will usually not work without altering the path, except when using pytest for testing. Pytest will itself alter sys.path so you find the modules you are looking for. If you don't want to use pytest then your test script needs to set PYTHONPATH before running python.

FYI: tracked by https://github.com/fable-compiler/Fable/pull/2578

Zaid-Ajaj commented 2 years ago

Python imports cannot reference out of their package/directory. Thus you cannot import e.g src from something in test without altering sys.path.

Given how common it is in dotnet to reference projects from sibling directories, I think it would be nice if the the compiled project just worked out of the box without having to know about sys.path, maybe have the compile the full project such that the entry point package is in the "root" and all other modules are subdirectories:

{outDir}
  | 
  | -- src
  | -- tests
  | -- program.py

Where program.py is the entry point because dotnet fable-py was run against the tests project. Note that in this case, the modules will no longer be compiled next to their F# sources. What do you think?

dbrattli commented 2 years ago

@Zaid-Ajaj I'm interested in anything that could lower the friction. Have been mostly busy with getting the all the bricks in place for compiling F# to Python, but very interested in input to make it easier to compile into Python. Will look closer into this tomorrow morning. Have btw published new fable-py and Fable.Core.Experimental so please check if that could bring your project one step further now that Union reflection should work a bit better: https://github.com/fable-compiler/Fable/blob/beyond/tests/Python/TestReflection.fs#L222

Zaid-Ajaj commented 2 years ago

Hi @dbrattli I've just updated the compiler and it looks better. There are a bunch of functions missing. I will try to list them here one by one later. I've updated the project now to make it easier to test it: basically removed the tests project and added an entry point to the library at ./src to avoid path resolution issues for the time being. This is of course not ideal because when I publish the library, I need to remove the entry point 😅 but it will do for now when testing the library

dbrattli commented 2 years ago

Hi @Zaid-Ajaj . Yes, I see that's quite a lot of work to get all those functions in place 😬 , but we should eventually get there: https://github.com/fable-compiler/Fable/pull/2580

dbrattli commented 2 years ago

@Zaid-Ajaj FYI: Have just published the fable-py.4.0.0-alpha-008.nupkg and Fable.Core.Experimental.4.0.0-alpha-008.nupkg packages, so please try again. This time I think you should hit the JS specific emits. If you can fix, then let's see what's next after that.

Zaid-Ajaj commented 2 years ago

@dbrattli Getting really close, I am trying to fix things locally and see if I get the project running. Here are the things that didn't work:

Missing these functions from fable_library.reflection.py
- get_generics
- get_function_elements
- get_tuple_elements
- get_enum_underlying_type
Missing the Lazy type from fable_library.util.py
Cannot use assignment syntax with Dictionary i.e. dict[key] <- value
For some reason, cannot use this type as key for a dictionary (error unhashable type)
Files fable_library.dateoffset.py and fable_library.boolean.py should use relative fable-library imports:
- from .types import FSharpRef instead of from fable_library.types import FSharpRef

I've updated the project again with more fixes and avoided things that I could do without (caching resolved types with dictionaries).

dbrattli commented 2 years ago

@Zaid-Ajaj Thanks for the feedback. Might take a few days for me to fix this (work week again) but I'm pretty sure this use-case will bring Fable Python to the next level. Fixes (wip) at https://github.com/fable-compiler/Fable/pull/2581

dbrattli commented 2 years ago

@Zaid-Ajaj For the Python port were impl F# Arrays using Python list. Python only have arrays on numeric types. So far using lists worked well, but for BitConverter it feels wrong to return a list of bytes when you really want bytes or bytearray since you are most likely going to send it somewhere. For Python what you usually would like to send would be bytes which is an immutable array (which we currently don't have in F#), i.e similar to ReadOnlyMemory<byte>.

It may also be complicated to use different backing of Arrays depending on the inner type. Any thoughts? @alfonsogarciacaro @ncave

F#	Python	Comment
List (F#)	List.fs	F# immutable list
List (C#)	`list`
Array	`list`	TODO: Python has arrays for numeric types
Map	Map.fs	F# immutable map
Record	types.py	Custom Record class. Replace with `dict`?
Option	Erased	F# `None` will be translated to Python `None`
An. Record	`dict`
dict	`dict`
Dictionary	`dict`	MutableMap if comparer
tuple	`tuple`
Decimal	`decimal`
DateTime	`datetime`
string	`string`
char	`string`

So a possible solution could be:

F#	Python	Comment
`[]<byte>`	`bytearray`	Python mutable str
`[]<sbyte>`	`array("b", [])`	Python array module
`[]<int16>`	`array("h", [])`	Python array module
`[]<uint16>`	`array("H", [])`	Python array module
`[]<int>`	`array("i", [])`	Python array module
`[]<uint32>`	`array("I", [])`	Python array module
`[]<long>`	`array("l", [])`	Python array module
`[]<uint64>`	`array("L", [])`	Python array module
`[]<float>`	`array("d", [])`	Python array module
`[]<single>`	`array("f", [])`	Python array module
`[]<'T>`	`list`	Python list module

Would something like this be very confusing for users? Then if you really want a Python list you should use C# List instead.

dbrattli commented 2 years ago

Did a quick check and native arrays actually seems to work really nice except for e.g Seq.sort depends on Array.sortInPlaceWith which is not supported for Python arrays. But this should be able to fix by moving the inner part of sortWith to e.g Native.fs:

let sortWith (comparer: 'T -> 'T -> int) (xs: seq<'T>) =
    delay (fun () ->
        let arr = toArray xs
        Array.sortInPlaceWith comparer arr // Note: In JS this sort is stable
        arr |> ofArray
    )

ncave commented 2 years ago

@dbrattli Pretty much the same as in JS, there are typed numeric arrays and untyped arrays.

dbrattli commented 2 years ago

@ncave The problem I now have is that modules such as Seq.fs and List.fs uses arrays as the backing store for many operations such as sorting, and I will need to use python lists (i.e ResizeArray) instead. Not I can do this without forking Seq.fs, List.fs. But it's probably fine when there is so many changes needed. Hard to make an abstraction that fits any target language. What do you think?

ncave commented 2 years ago

@dbrattli Yes, it's difficult to have the same exact F# library code for all languages. I know I have different implementations for Rust for some things (at least for now), as I need to work around missing features. As you can see for JS, some methods that need to make arrays for some purpose take (receive) an array constructor as an additional parameter, so the same code can handle different array types that differ by construction only.

dbrattli commented 2 years ago

@Zaid-Ajaj. Need to merge and generate new packages but looks promising!

Zaid-Ajaj commented 2 years ago

@dbrattli The test case worked?! This is HUGE 🤯 🤯 🤯 because it means the full reflection cycle has worked: type resolution, parsing, pattern matching and type activation.

Once the new bits are pushed, I'll start adding a lot more test cases similar to those in Fable.SimpleJson and see if we can iron things out even more

As for publishing the library, I think I will need to apply some gymnastics to the build script: removing the Program.fs from the library before pushing the nuget and adding it back once that is done to later test how a nuget package can be consumed (even tho we know Fable.Python works)

dbrattli commented 2 years ago

@Zaid-Ajaj Have pushed:

Please try and see if what issues surfaces this time. I'm sure there's still many things to be fixed, but I think it's really fun with the progress here 😄 I have btw not done anything with the --outDir issue yet to make sure the default compile have a common parent dir. This last PR btw increased coverage with ~100 additional passing unit-tests

Zaid-Ajaj commented 2 years ago

After modifying the implementation to work with Python better I have added a dozen or two of tests and here are the tests that are failing (along with their error message / missing imports)

Furthermore, using dotnet fable-py against any project without <OutputType>Exe</OutputType> creates invalid relative fable_module imports regardless of --outDir which means

Using <OutputType>Library</OutputType> creates an invalid project
Omitting <OutputType> altogether creates an invalid project (defaulting to Library?)

To avoid confusion, maybe it would make sense to output all the python sources into one directory such as ./dist or any given --outDir relative to the compiled project and fix the imports from there on?

dbrattli commented 2 years ago

@Zaid-Ajaj Thanks for the feedback. First let's discuss absolute vs relative imports. It's a tricky problem that I don't know how to solve in a good way. Did you btw read this? https://github.com/fable-compiler/Fable/issues/2577#issuecomment-955155692

The problem is that we need to use relative imports when building libraries. The library do not know where it's "mounted" e.g inside fable_modules when used as a nuget, or outside when using a project reference, or as a Python package (pip). There's also the Fable.Library.fsproj inside fable_library I cannot see how we can generate imports that are not relative, and still be valid. Let's look at an example:

mylibrary
     |--- __init__.py
     |--- util.py
     |--- library.py

Problem 1:

How should e.g library.py import foo from util.py?

from util import foo Not possible. This gives error ModuleNotFoundError: No module named 'util'. Only possible if library.py was a program (top module).
If it's mounted in fable_modules: from fable_modules.mylibrary.util import foo
If it's a project reference with a common outDir: from mylibrary.util import foo
Relative: from .util import foo (this works everywhere)

Problem 2:

A Program wants to use the library and import bar from mylibrary:

If it references mylibrary as a project reference with same outdir: from mylibrary.library import bar
If it uses mylibrary from a Fable nuget: from fable_modules.mylibrary import bar
Relative: ImportError: attempted relative import with no known parent package

Problem 3:

You want to augment a python lib you are writing with some F# code

This is similar to fable_library where we have Fable.Library.fsproj that generates some extra python files such as map.py, mutable_set.py.

Now e.g mutable_set wants to import FSharpRef from util.py. What import should we generate?

Relative: from .types import FSharpRef Absolute: from fable_modules.fable_library.types import FSharpRef.

PS: The absolute is correct (when used), but will look wrong in the editor, so harder to look at the generated code:

The problem is that we have no clue where this library will be mounted. For this example inside fable-library we could detect it, but for other libraries we have no way of generating a valid absolute import. E.g it could have been:

from abc.def.types import FSharpRef

Summary

We cannot generate valid absolute imports for libraries, we must use relative import
We must use absolute imports for a program, we cannot use relative imports

Currently we detect this using <OutputType>Exe</OutputType> vs <OutputType>Library</OutputType>. You say that setting it to Library (or not setting) creates an invalid project? How? Here are some examples where I know it works:

Perhaps one option could be to use absolute imports and have a command line argument to tell Fable its position in the library (e.g fable_modules.fable_library) and a command line argument to explicitly tell Fable to generate relative imports when we want it.

Or find a better way of detecting if we are a library or a program than using OutputType, but I currently haven't found any.

What do you think?

dbrattli commented 2 years ago

PS: Just pushed -alpha-010 which should hopefully fix your failing tests (and more) @Zaid-Ajaj

Zaid-Ajaj commented 2 years ago

Hi @dbrattli I think I understand the problem above and from my experiments I believe it only occurs when the entry point project (exe) references projects that are in sibling directories. If the entry point project (exe) is in a parent directory then referencing projects seem to work fine. Though I don't know what happens when the referenced/shared projects themselves have cross-references 🤔 I will need to investigate this.

The way I have worked around this problem for now in Fable.SimpleJson.Python is by turning the project into a library just before publishing to nuget 😓 it works but this shouldn't be the norm I hope when building libraries.

Good news though is that I published this thing and now it is working like a charm 🚀

simple-json-python

dbrattli commented 2 years ago

@Zaid-Ajaj Great work, and thanks for the feedback. I'll investigate this more in the weekend so hopefully we can find a better solution. I will also need to investigate:

The dictionary assignment issue. I think it must be related to Dictionary having a comparer, since it's then a MutableMap.fs and not a Python dict.
Unions cannot be used as keys for dictionary. Do you btw have the error message here?

dbrattli commented 2 years ago

@Zaid-Ajaj I think everything discussed in this issue have been addressed now so closing this issue. Please re-open a new issue if you find anything to not be working as expected.

fable-compiler / Fable