search

fable-compiler / Fable.Python

Python bindings for Fable
https://fable.io/docs/
MIT License
135 stars 10 forks source link

"static link" vs "dynamic link" #39

Closed thautwarm closed 2 years ago

thautwarm commented 2 years ago

I think you have thoroughly considered this issue according to https://github.com/fable-compiler/Fable.Python/issues/30#issuecomment-1003394317 , but let me repeat something here for being clarified.

Suppose we want to reference an external Fable Python library (namely, another Fable library not owned by us), there are at least 2 approaches and people might choose 2 different ways accordingly.

The first of which is "static link", we just take an external Fable library and generate code inside our package. Users do not need to install the Fable package. We keep the namespace of the external Fable inside our generated code, so that users can install that external package independently without polluting the environment.

The second of which is "dynamic link", we just take the type information of the external Fable library, and the generated code uses python import to access the library. The whole environment keeps only one version of the external Fable library.

I think both of them are useful. "static link" is easy to achieve, even without support from the Fable Python compiler. We can upload Fable project to any software registry and implement a tool to upload, download and verify libraries. For "dynamic link", I'm not sure if it is supported.

About the implementations.

For "static link", I think you have made progress and supported it in some degree, but I wonder if PyPI Fable packages are supported, if not I can help.

For "dynamic link", I wonder if Fable can extract type information that is usable for transpilation from the compiled dlls, I think "dynamic link" is do-able.

dbrattli commented 2 years ago

There are several use-cases:

  1. Fable project (library / app) referencing Fable compatible NuGet (F# bundled code). The NuGet may generate code that references additional PyPI packages
  2. Fable project (library / app) referencing Python library (PyPI). Use Fable.PyInterop namespace for making bindings similar to Fable.Python (similar to this repo).
  3. Python project (library / app) referencing compiled Fable library (uploaded to PyPI). Fable project needs to be packaged using e.g Poetry or similar.

We could add an option to the compiler to not bundle fable-library but reference PyPI installed library instead. For referenced Fable NuGet's this will be more difficult since it may not be trivial to convert them into a package. This was btw my first plan, but gave up and reverted to "static-linking" instead. One idea is to bundle a pyproject.toml with the Fable package so it knows how to install itself (dependencies etc). We could also have a compiler switch to generate Python packages instead i.e similar structure to:

poetry-demo
├── pyproject.toml
├── README.rst
├── poetry_demo
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_poetry_demo.py

Is it something like this you have in mind?

FYI: There is also Femto that handles NPM references for Fable (JS) NuGet's. We should extend it to support PyPI.

thautwarm commented 2 years ago

For referenced Fable NuGet's this will be more difficult since it may not be trivial to convert them into a package.

I think for NuGet packages we'd better keep them "static linking", before we get to know convert to them.

Is it something like this you have in mind?

Yes, but I'm not sure if bundling a pyproject.toml is sufficient. I was actually considering another format to control, (1) pyproject.toml, (2) .fsproj, (3) fable_modules (locally installed python packages and Fable packages).. but if we can avoid reinveting wheels that'd be great.

There is also Femto that handles NPM references for Fable (JS) NuGet's. We should extend it to support PyPI.

I don't have an idea about Femto, will check out this today. Initially I was considering to have a pyproject.toml for fable python projects to manage both fable python dependecies and pure python dependencies.

alfonsogarciacaro commented 2 years ago

Hi there! Not sure if I understand the problem well as I don't have much experience with Python packages (I just know they're very messy 😉) but I'll try to leave a couple of notes in case it helps. First, when you talk about a Fable library is this?

  1. A Nuget package that is Fable-compatible
  2. A Python package containing Python code generated by Fable

So far, in Fable JS when talking about Fable libraries it's about 1. Nuget packages. When Fable compiles a project it gathers all the sources from the referenced packages and treats them as if they belonged to the same .fsproj. We don't usually distribute precompiled code because there are not guarantees that generated code will be compatible with later versions of Fable.

However, it's possible that Fable packages also include some native (JS) code or make references to native npm packages. In the first case, library authors just put the .js files together with the .fs files in the fable folder included in the Nuget package, and Fable will just copy all contents of this folder to the output directory (within fable_modules) when compiling. In the second cases, library authors just use "bare modules" (ie. non-relative paths) that will be later resolved by JS tooling like Webpack.

The problem with "bare modules" is users often forget to install the npm dependencies and authors cannot specify version ranges either. This is what Femto tries to solve by reading metadata from the Nuget package so it can verify users have the proper npm dependencies installed.

Not sure if something similar can be done with Python. The ideal thing would be to avoid global package installations and have versions managed locally. Would it be possible to enforce this for Fable Python projects using tools like Poetry?

dbrattli commented 2 years ago

FYI: this issue will be fixed by https://github.com/fable-compiler/Fable/pull/2974 and #59 (if we decide to merge). With the suggested changes, fable-modules will contain proper Python packages, and you would have to use Poetry (or e.g change PYTHONPATH) to find the local modules. Fable compiled NuGet's should include a pyproject.toml file to list their Python dependencies. Projects also needs to have a pyproject.toml that refers to fable-modules (static) or PyPI (dynamic). Fable-modules will also be distributed to PyPI for dynamic usage.