fable-compiler / Fable

F# to JavaScript, TypeScript, Python, Rust and Dart Compiler

http://fable.io/

MIT License

2.86k stars 296 forks source link

MSBuild/NuGet integration: custom TargetFramework #3549

Open jwosty opened 9 months ago

jwosty commented 9 months ago

Description

There should be a Fable MSBuild SDK (read: https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-use-project-sdk?view=vs-2022) which provides tasks and targets to compile Fable code. If this were done, setting up a Fable project could be as simple as referencing the SDK, setting <TargetFramework>fable</TargetFramework>, then running dotnet build.

A proof-of-concept of exactly this grew out of a discussion we had in the FSSF Slack channel: https://github.com/jwosty/Fable.Sdk

This POC mainly answers the question: "Can we integrate Fable into the dotnet build process, with its own TargetFramework, and make it work with built-in MSBuild mechanisms (like package restoring)"? I think this shows the answer is: Yes. The main question left is whether or not NuGet package support is possible without NuGet having to know about a Fable TFM (see: section below)

Technical details

Some better integrations with the Fable compilation process would be awesome, and would allow for a much better implementation of this SDK. Currently, the POC accomplishes compilation through a FableCompile task which just runs the Fable CLI. For example, if Fable either exposed some of its internal functions which perform compilation publicly, or if the CLI allowed you to manually provide all the inputs (F# source files, package references, etc, a la fsc), then the SDK could completely bypass having to deal with an extra fsproj (no need for Buildalyzer when we have the real thing!).

Open questions

Below are some unanswered questions, as well as some design decisions that could go one of multiple ways:

Does NuGet.org allow custom target frameworks in packages? Or are we going to have to work with Microsoft to get it added to a hardcoded list for restoring to work?
Should the TargetFramework be completely standalone (i.e. fable), or should it extend a .NET TFM (i.e. net8.0-fable, like net8.0-mac? There are pros and cons to each.
Should the SDK concern itself with JS package managers (i.e. yarn, npm)? If so, how? Should it have special knowledge?
Should the SDK concern itself with JS bundlers (i.e. webpack, vite)? If so, how? Should it have special knowledge?
Assuming question 1 is resolved, how should packages be structured? Since we have our own TargetFramework, we can start to lean on existing NuGet folder conventions, but there would be many details to work out.
Should the SDK live in a separate repo, or here (https://github.com/fable-compiler/Fable)? The former is certainly possible, as long as Fable exposes the right hooks (be it new CLI parameters, or an API)
How do fable watch and hot reload (via webpack-dev-server for example) fit in?

ncave commented 9 months ago

@jwosty Can additional flags be passed to the dotnet fable tool? e.g. something like that:

<OtherFlags>$(OtherFlags) --lang TypeScript</OtherFlags>

jwosty commented 9 months ago

Question 3 is especially tricky.

On the one hand, there is some relationship between Fable versions and .NET SDK versions, because Fable implicitly supports a certain subset of APIs from some actual version of .NET (and lower). For example, .NET 8 will add various APIs which .NET 7 doesn't have, and Fable may choose to support some of these new APIs, so the Fable SDK would have to choose some .NET version to base itself off of.

On the other hand, say you're writing a project and you want to reference a Fable-compatible package. You really should only need to know the Fable version it targets; the .NET version which was used to built it is irrelevant, I think.

Either way, the tooling (mainly IDEs) needs reference assemblies to work properly. In the POC, I just turned those off completely, which is probably why intellisense is completely broken. So Fable really should define some reference assemblies, which it could do in one of a few ways:

Require at develop time that some particular .NET version is available (say Fable 4 requires .NET 7), and point to those reference assemblies
Bundle a particular .NET SDK's reference assemblies
Bundle its own reference assemblies (either derived from a .NET SDK, or created whole cloth)

In my opinion, this last one makes the most sense and is the most appealing, but is obviously also the most work. I have no sense for how much work but it seems big. But also awesome. If we did that, it might even make intellisense understand which methods are and aren't supported by Fable, at compile time!

jwosty commented 9 months ago

@ncave I did built in support for exactly that - OtherFlags is forwarded unmodified to the Fable compiler if you set it (see https://github.com/jwosty/Fable.Sdk/blob/7cb537fbd8092705274d6e887b23c7778e5d989f/src/Fable.Sdk/Build/Core.targets#L80).

However in terms of other target languages, I'm playing with overloading RuntimeIdentifier for that purpose. Currently it only lets you do <RuntimeIdentifier>javascript</RuntimeIdentifier>, but I imagine you being able to also do <RuntimeIdentifier>typescript</RuntimeIdentifier>

EDIT: Now that I think about this, I'm not sure whether or not that will be the correct approach. An alternative would be for there to be a TargetFramework for each langauge, for example fable4.0-js, fable4.0-ts, fable4.0-py (like net8.0-windows, net8.0-mac).

jwosty commented 9 months ago

Another point to investigate: Is it possible, and feasible, to make project references between non-fable (think: ASP.NET) and fable projects work? It's already possible in the vanilla .NET SDK to use ReferenceOutputAssembly=false to ensure a project is built without actually referencing its assemblies. Is it possible to build on this, to take a step further and reference its output items as content, rather than compilation input?

That being said, this is another problem which we don't have to solve right away, or even at all. It's a nice-to-have DX (developer experience) feature which could reduce friction for newcomers. We could just wash our hands of this complexity and just leave it as the user's problem.

MangelMaxime commented 9 months ago

Thank you for opening this issue and taking time to investigate on this feature.

It can seems obvious, but for such a feature to be included in Fable, it is important that the maintainers understand it. This is because it will probably fallback on us to maintain it and make it evolve over time.

I don't really know much about all the References, Runtime, MSBuild stuff.

The first time this feature was introduced to me, it seemed like the goal was to allow different packages to be used based on the Target language or if we target .NET vs Fable.

An example, for this situation is for example Thoth.Json and Thoth.Json.Net

However, to me it seems like this is complete new toolchain that is being proposed. Meaning that the user would not invoke Fable CLI anymore, but use MSBuild to configure a project which would in turn invoke Fable CLI or new entry point of Fable?

For example, if Fable either exposed some of its internal functions which perform compilation publicly, or if the CLI allowed you to manually provide all the inputs (F# source files, package references, etc, a la fsc), then the SDK could completely bypass having to deal with an extra fsproj (no need for Buildalyzer when we have the real thing!).

This kind of request is something that @nojaf asked about in the past.

My answer at the time, was that I didn't want to change how Fable CLI works yet. As it would split the focus and makes things more complex. But if we are do think about it of MSBuild integration this is also an opportunity to design it in a way that can be used for that.

@nojaf Could you please open an issue about what you wanted to do?

jwosty commented 9 months ago

@MangelMaxime Everything you're saying makes sense. If it helps, I'd be happy to own the SDK part of this and be responsible for updating/maintaining it. This could be a good reason for it to live in a separate repo, to make the boundary clear. Only minimal changes to this repo would have to be made.

The first time this feature was introduced to me, it seemed like the goal was to allow different packages to be used based on the Target language or if we target .NET vs Fable.

Yes, you are correct - if the full version of this proposal were completed, targeting would be very clear, since the .NET tooling would be made aware of this stuff. For example, instead of there having to be both a Elmish and a Fable.Elmish package, there could just be one Elmish, with two target frameworks - netstandard2.0 and fable4.0 (or fable4.0-js, depending on what we end up doing). This benefit flows downstream, too - consumers of Fable-compatible packages can just reference one package from one fsproj (multitargeting) and it will "just work," instead of having to fiddle with two separate but very similar project files. For example:

<!-- MyApp.Model.fsproj - would be able to be referenced both from .NET projects and Fable projects -->
<Project>
    <PropertyGroup>
        <TargetFrameworks>netstandard2.0;fable4.0</TargetFramework>
    </PropertyGroup>
    <ItemGroup>
        <Compile Include="Library.fs" />
    </ItemGroup>
    <ItemGroup>
        <PackageReference Include="Elmish" Version="X.Y.Z" />
    </ItemGroup>
</Project>

This is one of my biggest pain points when working with Fable (both apps and libraries), personally, so this would be a huge win.

A Fable user would also be able to tell canonically whether a package is Fable-compatible by looking at its target frameworks.

However, to me it seems like this is complete new toolchain that is being proposed. Meaning that the user would not invoke Fable CLI anymore, but use MSBuild to configure a project which would in turn invoke Fable CLI or new entry point of Fable?

Yes, that is one proposition. I'm mostly painting an overarching idealized vision, but we don't have to do everything all at once; it can be incremental. But yes, if this piece were done, the user would not have to invoke the Fable CLI to build a project; they would simply use dotnet build. We wouldn't have to get rid of the Fable CLI though -- it depends on how backwards-compatible we want to be. However this wouldn't be the first time that Fable has fundamentally changed its build pipeline (Fable 2 -> Fable 3, I think).

This kind of request is something that @nojaf asked about in the past.

My answer at the time, was that I didn't want to change how Fable CLI works yet. As it would split the focus and makes things more complex. But if we are do think about it of MSBuild integration this is also an opportunity to design it in a way that can be used for that.

For what it's worth, changing the CLI is not the only way. If I could reference Fable as a library, and had some function I could call to perform compilation given all of the compilation inputs (source files, project references, whatever else it needs), I could also make that work.

EDIT: I realized I already made this last point

jwosty commented 9 months ago

The more I think about it, the more I realize that the target language should probably be part of the TargetFramework rather than the RuntimeIdentifier. I'm not sure if NuGet takes the latter into account when resolving package compatability. If so, I favor RuntimeIdentifier. If not, a package intended for Javascript Fable wouldn't be able to prevent a Python Fable project from referencing it, so it'd have to be in TargetFramework.

If we go with TargetFramework, there should probably be a "root" fable4.0 which is compatible with all other TFMs, so that you could design packages which are designed to work with any target (presumably for packages which are pure F# and avoid runtime-specific APIs). A fable4.0-py project would be able to consume a fable4.0 package, but not vice versa. Likewise, a fable4.0-js project should not be able to consume a fable4.0-py project. This is similar to how .NET OS-specific TFM compatibility works (like net8.0, net8.0-windows, net8.0-macos, etc) (see: https://github.com/dotnet/docs/blob/main/docs/standard/frameworks.md#net-5-os-specific-tfms)

jwosty commented 9 months ago

Here's an interesting relevant discussion on WASM/browser TFMs on NuGet: https://github.com/NuGet/Home/issues/8186

nojaf commented 9 months ago

Hi there,

This is a very interesting proposal. Being able to re-use the NuGet restore from the SDK sounds really interesting. It could also be interesting if this worked with dotnet watch as well.

For what it's worth, changing the CLI is not the only way. If I could reference Fable as a library, and had some function I could call to perform compilation given all of the compilation inputs (source files, project references, whatever else it needs), I could also make that work.

Yes, this is what I mentioned in the past as well. Having Fable as a dotnet library would be beneficial. I'll create a new issue for that.

robitar commented 9 months ago

Really interesting! I've been working on a reasonably complex web app in Fable for the past year or so and have built a lot of complex Typescript/React apps before that.

I suppose firstly I should say that all these observations are based on building web apps in Fable, and treating it essentially as a replacement for Typescript in a front end web stack. I feel this is perhaps the best route for Fable to (continue to) take, because its such a good fit there, and the target runtime is reasonably well constrained.

I'm still somewhat unsure if I would use Fable in, say, a Rust project, There are non-zero costs involved, dealing with library bindings and non orthodox setups. I think I would rather just use Rust and benefit from the community etc. What makes it so compelling as a web tool, is that I know for certain that I would want to use it over JavaScript, even with all the burdens and overheads.

Anyway, some thoughts based on that and the setup I'm currently using..

Project layout There's definitely a disconnect between the 'dotnet' side and the 'npm' side, I essentially have two perspectives on the project (in VSCode). When I want to work on the 'web' side of things, I switch to the file explorer and it acts like a regular web project, when I want to work in F# I use the Ionide solution explorer. Getting this set up going is not entirely straightforward, so it would be good to formalize it somewhat, but I think you will always have some degree of separation here.
Bunders, npm etc I use esbuild and a few javascript files to run the build, nothing fancy. Those JavaScript files are invoked from npm scripts, for example: "start": "dotnet fable watch ./src -s --run node ./build/serve.js", where serve.js calls the esbuild api. This works well, is cross platform and is easy to maintain.

And some concerns:

If we try to make everything work in the msbuild file, will you have to start doing a lot of 'shell scripting in xml', or even worse, use custom msbuild tasks? I have nightmares from doing this in the old Team Foundation Server days, and would not like to go back there.
You might have a kind of 'Create React App problem', where its difficult or impossible to extend the build system to add some feature you need.
There's already a burden/overhead integrating with the web ecosystem, because you have to deal with bindings (the way it currently sits, you pretty much have to write your own for a tiny subset that you want to use etc). But at least you can use the industry standard tools to reference things, there's a package.json that just works, you can use npm/esbuld etc. If we move away from that, that it's just more and more alien and custom.

Anyway, all that being said, I think that having a custom target for Fable is a great idea, and being able to reference projects and perhaps even have multiple outputs (for doing stuff like web workers) would be great, but I think, at least for now, it's better to keep the division between the dotnet side and the messy 'web' side as far as tooling goes, and instead address that though guidance and templates.

So you could have a template for a 'Fable SPA Webpack' project, or a 'Fable Esbuild with Yarn' project etc. We could even look at going down the same route as Create React App, with a Create Fable App equivalent, which would provide a front end experience and generate and maintain artefacts like package and bundler configs for you. But you would have the option to 'eject' if you need to. The difference being that all of this stuff is not baked into the Fable tool, rather that would be just one part of the stack that CFA would manage for you, etc.

inosik commented 9 months ago

Related: #2338.

nojaf commented 9 months ago

Great insight @robitar!

I would like to add that when using Fable for the web, I really dislike that I'm forced to take the dotnet side of things. I need to compile to JS first and then run whatever is the relevant thing I actually want to run. There is this friction that you need to have your compiled JS on disk first in order for everything else to play ball. Many other file types that need processing don't take this approach.

In the example of esbuild, you probably want a plugin that deals with the F# files. That is how it would work with many other languages.

I have the same experience with Vite for example. Imagine using JSX, Sass, TypeScript and Markdown. All of these will be captured by Vite via plugins (or out of the box). There is no need to transform each file first, write it to disk in order that Vite knows what to do with it.

When using Fable, we always need to compile it first so that whatever comes after can pretend Fable doesn't exist.

While I for sure understand that creating a plugin for esbuild and Vite might be challenging, from a developer UX point of view I strongly believe this makes sense.

At the same time, I do very much see the appeal of what @jwosty is proposing here. Doing a dotnet build and having .js files will make a lot of sense for people.

I hope we should extract a lower-level component (see https://github.com/fable-compiler/Fable/issues/3552) so that both options can be explored by the community.

jwosty commented 9 months ago

Anyway, all that being said, I think that having a custom target for Fable is a great idea, and being able to reference projects and perhaps even have multiple outputs (for doing stuff like web workers) would be great, but I think, at least for now, it's better to keep the division between the dotnet side and the messy 'web' side as far as tooling goes, and instead address that though guidance and templates.

I would like to add that when using Fable for the web, I really dislike that I'm forced to take the dotnet side of things. I need to compile to JS first and then run whatever is the relevant thing I actually want to run.

This is actually really interesting, because I'm the opposite, personally - I dislike being forced to learn javascript bundlers since I come from the .NET side. Honestly I didn't consider the fact that some people may prefer the JS tooling - fair enough! XML is pretty gross.

I would hate to force people to drop down to MSBuild scripting when they're trying to avoid it, so how about this: we build this thing in more than one layer, making sure each layer stays very separate from higher layers. There'd be the lowest level - the Fable CLI (which already exists) and/or the Fable library (as proposed by #3552), which would allow for any kind of build solution you could come up with, be it webpack, make, FAKE, plain ol' shell scripts, etc. Then, there'd be a library providing just some MSBuild tasks, let's call it Fable.MSBuild.Targets, which would allow anyone to opt-in just a little bit to using the Fable compiler within their fsproj without having to go all-in on MSBuild (would look pretty similar to what I already do where I have a custom target with an <Exec Command="dotnet fable MyProject.fsproj" ... /> and a subsequent file copy). Third and finally, there would be the full-blown Fable.MSBuild.Sdk, which would probably be the right thing for anyone wanting to do as much stuff as first-class dotnet tooling as possible.

OK, so this addresses the build side. However, the other half of this suggestion is to improve the NuGet package creation and consumption story, and I'd love to hear your thoughts on that, too.

I'd still argue in favor of having custom Fable TFMs recognized by NuGet. In a perfect world, all Fable-compatible packages would have a target of fable4.0-javascript (or fable4.0 if they're not JS-specific). Unfortunately, installing such a package into your project would require that it also target fable4.0-javascript, which would require importing Fable.MSBuild.Sdk. Therefore all Fable packages would probably have to include a netstandard2.0 or net5.0 target, so that restoring still works for everyone. The Fable compiler (probably invoked via the CLI in these cases) would still work similarly to how it does today.

It's also certainly an option to make sure the Fable.MSBuild.Sdk has a way to only import it for the target framework and NuGet restore capabilities, without injecting its custom targets for Fable compilation.

jwosty commented 9 months ago

Also - just want to point out that it's probably important to decide whether or not we want at least the NuGet TFM, because I'm fairly certain we'll have to work with MS to make that happen. If @MangelMaxime approves, I'll be happy to file an issue over at NuGet/Home to get that ball rolling. (also @baronfel will be a help)

Here are some examples of past requests for new TFMs: https://github.com/NuGet/Home/issues/10800, https://github.com/NuGet/Home/issues/9347, https://github.com/NuGet/Home/issues/8186, https://github.com/NuGet/Home/issues/7773

robitar commented 9 months ago

I would like to add that when using Fable for the web, I really dislike that I'm forced to take the dotnet side of things. I need to compile to JS first and then run whatever is the relevant thing I actually want to run. There is this friction that you need to have your compiled JS on disk first in order for everything else to play ball. Many other file types that need processing don't take this approach.

Yea makes sense, especially when starting its quite confusing, and you are really on your own. However, with the setup I have, I never interact with dotnet directly (for packages I just paste directly into the project from the nuget listing etc). Its all just npm build and npm start.

In the example of esbuild, you probably want a plugin that deals with the F# files. That is how it would work with many other languages.

My big concern here would be performance. One of the main advantages of esbuild is that its so fast, it doesn't even show up on the console, even for a fairly large project (the output from Fable is about 800kb of JS). I'm also not sure how this would actually work, what would 'pull in' the F# project? An index.js somewhere?

I have the same experience with Vite for example. Imagine using JSX, Sass, TypeScript and Markdown. All of these will be captured by Vite via plugins (or out of the box). There is no need to transform each file first, write it to disk in order that Vite knows what to do with it.

I think the thing there is that these technologies use file based imports to build up the dependency tree, whereas F# relies on a separate project, which makes file by file inclusion tricky, you definitely wouldn't want to be parsing the project for each file. There was a language request a while back to move F# to an 'import' style model, but obviously that might be a long way off it ever.

At the same time, I do very much see the appeal of what @jwosty is proposing here. Doing a dotnet build and having .js files will make a lot of sense for people.

I hope we should extract a lower-level component (see #3552) so that both options can be explored by the community.

Yes agreed, having some solid underling technology would be great, something akin to Script# - you take a dotnet based project and instead of outputting IL it outputs JS, and has some reasonable tooling out of the box for minification/watching, but if you want to use industry standard stuff for that, you can.

jwosty commented 9 months ago

Yes agreed, having some solid underling technology would be great, something akin to Script# - you take a dotnet based project and instead of outputting IL it outputs JS, and has some reasonable tooling out of the box for minification/watching, but if you want to use industry standard stuff for that, you can.

Keep in mind that it's possible to use dotnet tooling for the initial F#->JS compilation, then a JS bundler after that. For example, in this sample project (from Fable.MSBuild.Sdk), you invoke the build at the highest level with dotnet build, which ends up calling the Fable compiler and emitting the output in a path inside bin/, then calls webpack, which does its thing. You can do this even today (without my SDK).

Also, I'm taking that exact philosophy of ".NET is to IL assemblies as Fable is to JS" while designing the SDK.

There should probably be some more samples out there demonstrating these other ways to set up builds. It really is not so clear when you're first learning.

robitar commented 9 months ago

This is actually really interesting, because I'm the opposite, personally - I dislike being forced to learn javascript bundlers since I come from the .NET side. Honestly I didn't consider the fact that some people may prefer the JS tooling - fair enough! XML is pretty gross.

Well just for the record I don't actually 'like' using all the npm stuff 😄 and it was a slog to learn it over the years, this is a matter of pragmatism. If you are building a real-world app, after a certain point, you can't pretend like all this stuff doesn't exist, unless you like reinventing wheels all the time!

When I started out down this road, I too wanted to ditch npm and friends, and even experimented with a tool which would automatically pull in packages and produce ts2fable bindings on the fly. But completely supplanting the npm ecosystem like this is not straightforward. It can surely be done, but its going to be a significant undertaking.

There's already a reasonable penalty to using Fable - you are off the beaten track and you need to deal with writing your own bindings. F# is worth it for me, no question, but I'm not really going to turn my back on the entire ecosystem either. Using the standard tooling means you have one less thing to translate and manage.

I suppose I would also just say, on a more philosophical note perhaps, that I think it's better to actually accept that you are indeed building a web app, using web technologies etc, and not try to sweep this under the carpet. It's probably easier for me, I started out in web and I've never 'hated' JS/CSS etc, even in the dark days, although it was certainly a lot hacker than it is today!

The way I see it is: Fable is a way to use F# to make browser apps based on native web technologies, which is a slightly different proposition to: Fable is a way to convert F# to JavaScript.

Once I accepted this, I knew where I stood. I think this distinction is even more relevant now that you have Blazor/Bolero. To my mind, it makes the value proposition even clearer, and more distinct. If you want a full dotnet experience, use Bolero, if you want to deeply integrate into the web ecosystem, use Fable.

I would hate to force people to drop down to MSBuild scripting when they're trying to avoid it, so how about this: we build this thing in more than one layer, making sure each layer stays very separate from higher layers. There'd be the lowest level - the Fable CLI (which already exists) and/or the Fable library (as proposed by #3552), which would allow for any kind of build solution you could come up with, be it webpack, make, FAKE, plain ol' shell scripts, etc. Then, there'd be a library providing just some MSBuild tasks, let's call it Fable.MSBuild.Targets, which would allow anyone to opt-in just a little bit to using the Fable compiler within their fsproj without having to go all-in on MSBuild (would look pretty similar to what I already do where I have a custom target with an <Exec Command="dotnet fable MyProject.fsproj" ... /> and a subsequent file copy). Third and finally, there would be the full-blown Fable.MSBuild.Sdk, which would probably be the right thing for anyone wanting to do as much stuff as first-class dotnet tooling as possible.

Agreed. I think this very reasonable.

OK, so this addresses the build side. However, the other half of this suggestion is to improve the NuGet package creation and consumption story, and I'd love to hear your thoughts on that, too.

I'd still argue in favor of having custom Fable TFMs recognized by NuGet. In a perfect world, all Fable-compatible packages would have a target of fable4.0-javascript (or fable4.0 if they're not JS-specific). Unfortunately, installing such a package into your project would require that it also target fable4.0-javascript, which would require importing Fable.MSBuild.Sdk. Therefore all Fable packages would probably have to include a netstandard2.0 or net5.0 target, so that restoring still works for everyone. The Fable compiler (probably invoked via the CLI in these cases) would still work similarly to how it does today.

The TFM looks good to me, it would also mean, presumably, that you would have a more robust way to find Fable packages on Nuget directly, instead of having a separate package index.

I probably don't fully understand the drawback there, is this because you may want to have a single project which includes Fable and regular CLR packages?

robitar commented 9 months ago

Keep in mind that it's possible to use dotnet tooling for the initial F#->JS compilation, then a JS bundler after that. For example, in this sample project (from Fable.MSBuild.Sdk), you invoke the build at the highest level with dotnet build, which ends up calling the Fable compiler and emitting the output in a path inside bin/, then calls webpack, which does its thing. You can do this even today (without my SDK).

Also, I'm taking that exact philosophy of ".NET is to IL assemblies as Fable is to JS" while designing the SDK.

This makes sense, and I think would be a great bedrock, I guess I'm overepmhasising the web nature of things, there are certainly non-web JS environments and having a sensible and well ordered tool chain for that would be very nice.

I've mentioned Script# a few times here, looks like its still around, sort of: https://github.com/nikhilk/scriptsharp

I used this way back in the day, and it covers more or less the same ground. It manifested as a regular C# project, and leaned on all the integrated .NET Framework tooling (including Visual Studio). It output both dlls and script, so you could reference one library in another, which made the workflow feel exactly like a regular CLR build.

I think we can take that example and apply it more or less to Fable and hopefully get similar results. It would be nice to have multiple projects organized into a solution, some Fable, some not, reference packages (some Fable, some regular) and have everything just work out of the box!

There should probably be some more samples out there demonstrating these other ways to set up builds. It really is not so clear when you're first learning.

Absolutely. If I were new to web development, or not strongly motivated to use F# etc, I wouldn't have stuck with it at all. There are a few samples I think, but what's really lacking is real world stuff that's up to date. I'd be happy to help out there perhaps, with a sample of a real world end to end toolchain, although it probably makes sense to see what we want to aim for/change first.

nojaf commented 9 months ago

My big concern here would be performance. One of the main advantages of esbuild is that its so fast, it doesn't even show up on the console, even for a fairly large project (the output from Fable is about 800kb of JS). I'm also not sure how this would actually work, what would 'pull in' the F# project? An index.js somewhere?

The plugin would probably need to have the project as input. However, would only need to be processed once and I don't think this is any different from how it works today. I don't overly see performance as the bottleneck just yet. It will bring its own slew of challenges for sure.

jwosty commented 8 months ago

Just want to note that there are two main possibilities for a TFM design:

Option A: (Fable + Target lang) as TFM

This would look like: fable4.0, fable4.0-js, fable4.0-py, etc

Pros:

Decouples Fable versions from .NET versions

Cons:

It would be a new special corner case in the .NET world

Option B: (.NET TFM) + (Fable + Target lang)

This would be the MAUI-style naming, like: net8.0-windows, net8.0-windows7, net8.0-macos, etc. Tizen also used this, and the GtkSharp SDK is going this direction.

This would look like: net8.0-fable, net8.0-fable4.0, net8.0-fable4.0-js, net8.0-fable4.0-py, etc

Pros:

Might require fewer .NET ecosystem changes to work, because it can piggyback off of existing .NET SDK TFM rules (this may even almost work today)

Cons:

Would introduce a relationship between Fable versions and .NET SDK versions - i.e. why should it matter that this Fable 4 package was built with the .NET 5 SDK?

Either way, the compatibility rules would looks something like:

Language-specific TFMs (fable4.0-js/net8.0-fable4.0-js) would be able to reference the generic TFM (fable4.0/net8.0-fable4.0)
Generic TFM would NOT be able to reference language-specific TFM
Potentially, Fable TFMs should be able to reference with warnings certain vanilla .NET TFMS, for backwards-compatibility with "legacy" Fable packages
- If option A, which vanilla TFMs? net5.0 and upward? netframework4.X? Or just netstandard2.0 and anything transitively compatible and be done with it?
- This would produce a warning similar to what you get when referencing a .NET framework package from a .NET Core project

And perhaps an additional compatibility rule allowing anything to reference a Fable project as content? Though that might be more an SDK concern than a NuGet concern, and I'm not sure on the specific mechanisms, but that could be interesting to investigate.

MangelMaxime commented 8 months ago

Here is a summary of our discussion

@nojaf @jwosty Please feel free to complete or correct me

NuGet integration

We would not go with Option B because currently, Fable is not tied to a specific version of .NET SDK.

The convention is that Fable always support a subset of the .SDK/BCL APIs.

The available APIs are controlled by the target framework / packages used. And then Fable will either succeed in replacing it or print an error message. And we can add support for more BCL APIs in a minor version of Fable.

With that said, we would probably go with Option A but by using full language names for consistency like:

fable4.0-dart
fable4.0-dotnet
fable4.0-javascript
fable4.0-python
fable4.0-rust
fable4.0-all

We can contact the NuGet team, not to ask them to add them right now. But more to have guidance of if this is possible and what are the requirements / drawbacks.

Example of questions we have for them or internally:

Do we need to provide an exhaustive list of all the Fable TFM?
Can we provide only a pattern for the Fable TFM?
How long does it take to add new TFM support?
If we go with TFM, does it impact how Fable works currently? Does it block us if we want to work on Fable 5 in the feature?
Do we need corresponding compilers directives?
How does it play with backward compatibility? For example, currently we can compile using .NET 8 while targeting and older framework like netstandard2.0.

@jwosty as you seem more familiar with the NuGet/MSBuild stuff, would you be ok to contact the NuGet team?

MSBuild integration

We will need to define what the goal of that integration is? Opinionated "framework" to work with Fable?
Make sure the MSBuild Fable SDK works with IDE out of the box.

robitar commented 8 months ago

NuGet integration

We would not go with Option B because currently, Fable is not tied to a specific version of .NET SDK.

Although this may be technically true, I don't think this is semantically right. Fable does track against F# language features and types which change with dotnet versions (types such as DateOnly for example). You also take a kind of practical dependency on the dotnet version tooling, as in, your IDE thinks this is a regular F# project running on dotnet 7 etc.

It's a subset, yes, but its still a strongly coupled subset. I think this is more in keeping with the spirit of platform specific TFMs. The Fable runtime is just an extremely constrained platform.

robitar commented 8 months ago

I would be leaning more towards Option B, for the reasons outlined by @jwosty but also as mentioned above, it seems more practical to acknowledge that there is a clear coupling at least in terms of tooling, but also in terms of language features.

I would prefer to see Fable become more 'normalized' in the dotnet ecosystem, and it seems that setting it up as this weird isolated thing with no relation at all to dotnet proper would be moving in the opposite direction.

MangelMaxime commented 8 months ago

Problem with option B, is that it requires new TFM for each .NET release and makes documentation and authoring library more complex for author libraries. For example, I am using F# since 7+ years and don't understand all the implication that NuGet integration yet.

In any case, the questions we plan to would be the same if we go in this direction. For now, we are mostly exploring ideas, feasibility, features and drawbacks if we were to add TFM for Fable.

It is also important to note that TFM support here is mostly to have the ability to allows Fable libraries to have different dependencies depending on the targeted runtime (JavaScript, Python, etc.). It is not really coupled to which version of Fable you use are using to compile the project. This is still control by the user when installing fable CLI tool.

jwosty commented 8 months ago

It is also important to note that TFM support here is mostly to have the ability to allows Fable libraries to have different dependencies depending on the targeted runtime (JavaScript, Python, etc.). It is not really coupled to which version of Fable you use are using to compile the project. This is still control by the user when installing fable CLI tool.

For what it's worth, it probably would control the version of the Fable tooling used, or at least have some relation to it. If we do both NuGet and MSBuild integration, the Fable user wouldn't need to install the CLI tool at all. Referencing the Fable SDK instead of Microsoft.NET.SDK would bring in the stuff it needs for building; see for example this example project:

<Project Sdk="Fable.Sdk/0.0.1">
  <PropertyGroup>
    <TargetFrameworks>fable4.0-javascript;fable4.0-typescript</TargetFrameworks>
    <NoWarn>NU1701</NoWarn>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="App.fs" />
    <Content Include="webpack.config.js" />
    <Content Include="public\fable.ico" />
    <Content Include="public\index.html" />
  </ItemGroup>
</Project>

Obviously it doesn't prevent you from also installing the CLI tool; it's just that you wouldn't need to

robitar commented 8 months ago

Yea having a 'real' SDK setup seems much more useful to me, rather than just having a smart tag to filter nuget references (which is kind of useful, but probably wont have as much impact on fostering Fable usage as this would).

It also touches what I was trying to express above. Even if there isn't a runtime relationship with the CLR/dotnet, there's a definite tooling/dev dependency with the dotnet SDK, and it seems that having a proper integration at that level is actually more relevant than trying to define the relationship with Fable/CLR in terms of target platforms (although that is still beneficial, it's a more complicated knot to untangle).

jwosty commented 8 months ago

That's actually a good point @robitar - any custom MSBuild SDK which hooks into concepts from the .NET SDK (like target frameworks) implicitly have a dev-time dependency on it. This Fable SDK wouldn't make sense in the absence of the .NET SDK, and probably a particular version, too. I'm not sure what guarantees .NET makes in terms of backwards-compatibility of MSBuild targets, but at the very least it would probably implicitly depend on a minimum version of the .NET SDK.

Though once again this may be more of a concern of the actual SDK itself, rather than packages built with the SDK. Consider this hypothetical: Say there's (at least) two versions of the Fable 4 SDK, one which works with .NET 7 and one with .NET 8. Also, say we're authoring the Elmish package, and we happen to be using the .NET 7 one. A downstream consumer building an app which references Elmish probably shouldn't need to know nor care about that, right? I see no reason they shouldn't be able to use the .NET 8 one for their app.

baronfel commented 8 months ago

backwards-compatibility of MSBuild targets

Broadly, as long as you're interacting with a Target, Item, or Property whose name doesn't start with _ you should be fine - especially if that thing is in the MSBuild Common Targets or the 'base' .NET SDK.

jwosty commented 8 months ago

Alright, I've started a thread to ask about a NuGet TFM: https://github.com/NuGet/Home/discussions/12965

nojaf commented 8 months ago

@jwosty how would one download the Fable SDK <Project Sdk="Fable.Sdk/0.0.1"> in the first place? Is that something that NuGet does as well?

jwosty commented 8 months ago

@nojaf the statement itself downloads it. https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-use-project-sdk?view=vs-2022#how-project-sdks-are-resolved