search

fable-compiler / Fable

F# to JavaScript, TypeScript, Python, Rust and Dart Compiler
http://fable.io/
MIT License
2.92k stars 300 forks source link

[All languages] - Universal package management approaches #3089

Closed alexswan10k closed 2 years ago

alexswan10k commented 2 years ago

With the Rust/Dart/Python targets becoming less immature by the day, it has got me thinking... Perhaps it is time to revisit the package management/distribution debate in the light of an approach that could work for all targets.

I am not an expert on any of this, so hoping those that know these systems inside-out will be able to chime in and correct me where I am talking rubbish! I will go back and update this post as new info comes in.

State of the world today

Nuget packages ship binaries only by default. This is not compatible with Fable. In order to make a library compatible, F# sources need to be included, which can then be compiled by the consumer. This is somewhat of a manual process, but there is an interesting discussion on improving this here https://github.com/fable-compiler/Fable/discussions/2939. Discoverability is also quite limited

https://fable.io/docs/your-fable-project/author-a-fable-library.html https://fable.io/community/ https://fable.io/resources.html#examples

Previous discussions

https://github.com/fable-compiler/Fable/issues/1649 https://github.com/fable-compiler/Fable/issues/856 https://github.com/fable-compiler/Fable/discussions/2939

I should also mention Femto at this point, which already bridges the impedance mismatch between ecosystems (npm/yarn vs Nuget/Paket) in order to create a more unified experience. This approach seems to be valid for bindings.

A brief summary of target ecosystems

Each compilation target generally has its own ecosystem for managing dependencies.

Some other hypothetical examples that may help generalize the approach

Frustrations with Nuget/Paket and the .NET ecosystem

One of the core pain points many developers hit eventually is the transitive dependency collision problem. This happens when you have something akin to the following dependency tree:

Due to the fact that nuget is heavily opinionated on how the .NET CLR loads dll's, it forces the world-view that all libraries are global, and thus there can only ever be 1 source of truth when referring to a namespace/type. Although this is less of a problem in F# because dependency tree's are generally less deep, this seems to me to be a fundamental flaw which is unlikely to ever have a good solution at the .dll level. This puts limits on the complexity of libraries, and their ability to reuse existing code.

More modern ecosystems (such as js - npm and Rust - cargo) get around this problem by supporting scoped, or aliasing of package names, and referring to packages in code by their aliased name.

Questions

MangelMaxime commented 2 years ago

Hello @alexswan10k,

I don't understand what you are suggesting.

Something important with Fable is that it live inside of F# ecosystem and Fable is not an independent language. For example, Fable doesn't really compile or resolve dependencies (I simplify I know). It is the F# compiler which does it, then generate the AST and Fable kicks in from this step.

Distributing the Fable libraries via NuGet allow you to have access to all the F# tools for free: IDEs, analysers, linters, formatters, etc.

Currently, there are mainly 2 scenarios when distributing Fable libraries:

I don't see why targeting another language other than JavaScript would not work the same way it does with JS. Fable 3 and 4, emits files on the disk so it is as if you wrote them manually.

It is true that if one of the library you use depends on a target libs (like React, Express, etc.) then you need to install both the Fable NuGet + the target lib via the your preferred dependency manager (NPM, Yarn, Cargo, LuaRocks, etc.).

Having our own dependency manager would mean having to fork or adapt the existing F# ecosystem, change how F# works so have a custom compiler, make adaptation to IDEs, and a dependency manager is a really complex piece of software to create and maintain. Especially if you want to be multi target.

In the past, deciding to embrace the JavaScript ecosystem tools and standard has always proved to be of a huge benefit for Fable. And I do think, the same will be true for others languages.

IHMO the current position of Fable, allow it to benefit from both F# tooling and target ecosystem tooling for "free". And allows, us to focus on only the specific job of Fable which is translating the F# to language X.

MangelMaxime commented 2 years ago

If a library is written using pure Fable compatible code it will works across all the target language the same way.

For example, if my library consist of:

module SimpleMath

let add a b = a + b

and I ship it to NuGet.

I will be able to consume it for JavaScript, Rust, Python, etc the same way.

If needed compiler directives can be used to make adaptation depending on the target language:

let log msg =
#if FABLE_JS 
    console.log msg
#endif

#if FABLE_Python
    print msg
#endif
alexswan10k commented 2 years ago

Hi @MangelMaxime,

I do apologise I think my knowledge on this may be a little stale, as this has flip-flopped a couple of times over the lifetime of Fable. I have updated the original ticket to reflect your observations.

So it does appear that F# code can just be distributed with a Nuget package, and compiled by the consuming system on demand (a little similar to the Cargo approach, but without internal knowledge of modules). Assuming the target language has working module support, this should just work out of the box as you pointed out. This is a pretty good place to be if so.

I guess maybe the real question then is around refining the user experience of authoring and finding these packages as mentioned here. It was not obvious to me where nuget libraries support or do not support Fable (and which would be language agnostic), which is a fundamentally different ecosystem than .NET.

MangelMaxime commented 2 years ago

@alexswan10k

I do apologise I think my knowledge on this may be a little stale, as this has flip-flopped a couple of times over the lifetime of Fable. I have updated the original ticket to reflect your observations.

No problem, it is true that it changed several times. The NuGet distribution is here since 3 years I think and has been pretty stable since them.

Can the bar be lowered for package author's to make their packages Fable compatible

I still think that not much can be done on this side. Right now creating a Fable package is just about adding:

<!-- Add source files to "fable" folder in Nuget package -->
<ItemGroup>
    <Content Include="*.fsproj; **\*.fs; **\*.fsi" PackagePath="fable\" />
</ItemGroup>

to the *.fsproj.

For pure bindings, nothing special is required.

Is there a way to improve visibility on which packages are compatible with Fable, and if so are they constrained to 1 target?

There has been some initiative in the past:

However both of them require a manual editing and UX is not that great.

IHMO, the good solution would be use tags when authoring Fable package:

<PackageTags>fable;fsharp;json;fable-js;fable-python</PackageTags>

And then have a website/tool which send search request to NuGet.org server. This means that if we want to list all Fable package we can search for tag fable. If we want to limit to JavaScript Fable packages fable-js is the tag to search for etc.

Is there a solution to the transitive dependency problem, is it even important?

I would vote for "not important" because I don't think I encounter this problem in the past. 🤞

alexswan10k commented 2 years ago

Thanks for this, it fills in a lot of gaps. It looks like most of my concerns basically have solutions today, it's the age-old documentation problem of "you cannot search for what you do not know about" I guess. I noticed a lot of these details are documented too, which is great. I just had trouble finding them. I could not find https://fable.io/community/ though from the main site.

The tagging/nuget api query on tag solution seems like an excellent idea to me.

MangelMaxime commented 2 years ago

About https://fable.io/community/ it never really worked and when merging "Awesome Fable" with the rework of fable.io Alfonso and me kind of agreed on making it deprecated.

Because, it would mean that we need to maintain the list of items manually in 2 different places and also it doesn't fit the look right now and would require work etc.

IHMO, if something is to be done or happen it is to experiment with the tagging/nuget API. Because, this would also a single source of truth without any work require from the maintainers. And also, the experience can be deeply customized if needed.

ncave commented 2 years ago

Thank you @alexswan10k for your questions and thoughts, and thank you @MangelMaxime for your detailed response! I absolutely agree with what @MangelMaxime said about Fable benefiting from both F# and target ecosystems for "free". I believe that was what @alfonsogarciacaro's goal was from the very start.

My personal opinion has always been that Fable should just be a transpiler, and as such, stay within the design goals that other transpilers like TypeScript have, as stated in their "non-goals" (i.e. their "we don't want to be that" section), more specifically "non-goals" 4 through 7, which IMO can be adopted verbatim in Fable documentation.

alexswan10k commented 2 years ago

Thanks, both, and I do agree with reusing the F# ecosystem where possible etc. I guess I am also just aware that there are limitations due to external constraints such as .NET dll loading behaviour (everything global), which will forever put F# at a disadvantage against its competitors that have a more modern package management philosophy (locally scoped packages eg npm or Cargo). This is absolutely not a Fable problem though, it just turns out Fable is not theoretically constrained by these shackles, which is an opportunity in my view.

Anyway, nuget querying seems like an obvious place to improve matters. Perhaps I can take this away and have a look at crashing out a page for integrating with https://api.nuget.org/v3/ for the fable.io site. I imagine this should be pretty straight forward, and can probably be done through the public api, and thus be entirely client side. I will report back..

alexswan10k commented 2 years ago

Looks like it is possible https://stackblitz.com/edit/react-ts-d8llqx?file=Search.tsx https://react-ts-d8llqx.stackblitz.io

Nothing is using tags at the moment, so we might want to fallback on using query as a stopgap.

I will put some UX fluff around it anyway. How would one host something like this in the core site? It looks pretty much markdown-only from a cursory glance. Can we run react components?

Edit - Looks like you cannot query tags according to the api docs. You can query packageType though. Is this an option?

ncave commented 2 years ago

@alexswan10k I don't see why not, but existing Fable packages will have to be republished with a package type so they can be discoverable this way. Also, I don't see package type listed anywhere in the existing UI on nuget.org or nuget.info.

MangelMaxime commented 2 years ago

For hosting, it can be done two ways.

Either inside the current site, because we can generate HTML or load JavaScript on a specific page. For example, this page from Fable.Form is just using the doc generate to have the navbar generated and all the page content is actually an SPA.

Or, we can create a new repo inside of fable-compiler org to host it separately. For example, https://fable.io/packages.

The benefit of the first solution is that the navbar and style are handled by Nacara (doc generator) and so it will easy to have the same look and feel between that and this specific page.

The cons is that the repo will not contains simple documentation anymore but also the source code for that search packages page.

IHMO, it is better to host that specific page/section under it's own repository. It will make it easier to accept contribution, prototype stuff, and if necessary deviate from the standard styles to improve the UX. We just need to use the same color and basic style to have a consistant look and feel.

alfonsogarciacaro commented 2 years ago

Thanks everybody for your comments! Yes, as @MangelMaxime and @ncave say we should likely avoid trying to build our own package manager and reuse existing infrastructure instead. Package management is one of these deceiving problems that are much more complicated than they look at first, as Paket maintainers can corroborate ;) However, if we can do something to improve the experience of Fable users, particularly around discoverability as @alexswan10k says, that'd be great.

Some notes:

alexswan10k commented 2 years ago

Thanks all for the input.

I have been pushing forward the discoverability angle, and I think I pretty much have a working search page (using tags after all) that talks to the nuget api. The big problem now is going to be agreeing on all the tags, and getting package authors to correctly follow the convention :)

I propose the following convention for tags:

fable-target:all - No proprietary code. Should compile for all future targets as long as language features supported
fable-target:lang (eg js/dart/python) - Has proprietary code, usually Emit statements for interop etc.

This convention could also be extended to cover bindings

fable-binding:js
fable-binding:php

Here is the working page: https://react-ts-d8llqx.stackblitz.io

Editable: https://stackblitz.com/edit/react-ts-d8llqx?file=Search.tsx

image

I crossloaded the bulma stylesheet from fable.io by the way, so perhaps not the most elegant, but it gets the same component styles and colour scheme etc for free. It is pretty rough around the edges right now, but hopefully, the intent is self-explanatory. We can always do more polishing down the road.

Happy to take direction from here, let me know. Should I make a repo?

MangelMaxime commented 2 years ago

I find that -target suffix add verbosity without much gain compare to just fable.

Personally, I would go with:

-binding suffix is good to me. We just need to use the same lang convention.

UX related:

  1. It would be nice if the logo container size could be consistent.

See I think we should decide on a height and make the logo adapt the best it can to that height.

  1. Blue text over blue background is difficult to read.

This is fine for the first text "This will search Nuget.org for any packages with the following tags" but for the main content I don't think this is good.

I wonder if we could use a similar style as for the Blog list for the boxes or if that would be too heavy.

alexswan10k commented 2 years ago

Updated as per feedback

https://github.com/alexswan10k/fable-nuget-search

alfonsogarciacaro commented 2 years ago

FYI: Femto already supports Python packages :tada: https://twitter.com/zaid_ajaj/status/1567101357470978053

MangelMaxime commented 2 years ago

Thanks to the work done by @alexswan10k I think we can say that it is possible to search for packages using the NuGet API.

We can move the discussion to https://github.com/fable-compiler/packages

I created an issue with a specification proposition: https://github.com/fable-compiler/packages/issues/1

About the design/UX of the tool itself, I have some ideas how we can improve what has been started by @alexswan10k.

alexswan10k commented 2 years ago

Great. Thanks @MangelMaxime.

Do let me know if you need me to move anything, transfer access, host, or whatever.. Looks like it the target repo is empty? Maybe it is just that old eventual-consistency thing :) Will check back in a bit.

MangelMaxime commented 2 years ago

I just created the repo to have a place dedicated to that tool and especially for the hosting the specifications etc.

I have not started yet to upgrade/write the tool itself yet.