There has been a long standing desire to add the ability to reference nuget packages from F# scripts. Originally this was conceived as basic fully qualified nuget references like packages.config. Lately this has evolved into integrating "dotnet" or "paket" or "npm" package specifications and dependency management tools as part of the toolchain, or providing generic hooks to allow this.
Also package references in Python, R etc. scripting
Design principles
You can add a package reference to a script with a single line using a normal text editor that supports F# - no extra files (e.g. a packages.config) are needed
Package references include version constraints, and dependency resolution is performed a.la. nuget v3 and/or paket
it works "at design time" , i.e. I can open a script containing package specifications and quickly get editing and type checking against a resolved set of packages - without needing to run any of the script or any command line tools
it works universally, i.e. it works on any default install of F# editing + scripting, whether Ionide or the Visual F# Tools or web-hosted delivery of F# scripting such as Azure Functions or Azure Notebooks.
It aligns well with how package management will be dealt with in the future of the .NET toolchain
it works "the same way" on both .NET Framework/Mono and .NET Core, at least to some approximation,.
It considers the needs of F# when used as a Javascript programming language through toolchains such as Fable or WebSharper. Here NPM is a natural package manager, though there are others.
The design and implementation do not induce a dependency on any one specific package manager within the core F# toolchain (i.e. compiler/scripting/editor/Fsharp.Compiler.Service/ProjectCracker). It might be that different package managers have some specific support to make them work, but we remain open to new package managers.
The implementation does not induce bad "layering problems" in the F# toolchain implementation reminiscent of MSBuild, see below.
It works efficiently - package resolution is amortized, for example.
The default settings in editing and execution tools are sufficiently space-efficient, sharing packages between scripts if needed to achieve this.
Possibiity. "dotnet" references in scripts
The way the new dotnet core tooling loads nuget packages and their assemblies is awesome! I've been using its extensibility to build a DotnetCliTool. All the dependencies are downloaded & loaded from the single %userprofile%\.nuget\packages directory. I would like to be able to use this same mechanism from scripts. I would like the same types of reference to be supported, package references and project references.
Instead of having to use a nuget client to download the package and then reference the assembly like:
Possibiity. #project "my.fsproj" references in scripts
People have suggested that project references should work the same way as package references to work in Visual Studio 2017 with the dotnet core tooling. e.g. be able to do:
#project "my.fsproj"
Possibiity. #r "paket: Foo.Bar.dll" references in scripts
The experiment https://github.com/Microsoft/visualfsharp/pull/2483 contains a prototype of integrating paket package management directly into the F# programming model for .NET Framework programming. This includes design-time support. The experiment violates one of the design principles above - it "bakes in" support for Paket only. However that support could be factored into either fsi.exe.
Possibility. #r "packages" references in scripts
People have suggested that a script simply referencing #r "packages" should implicitly pick up the packages from its context, e.g. the containing packages.config or solution or paket.dependencies in the toolchain. In the Experiment adding Paket support to FSI this is #r "paket"
Possibility. implicit generation of load scripts
Paket has a feature to accurately generate .fsx and .csx load scripts containing the #r references suitable for use with F# and C# scripting. This feature is a natural and simple way to integrate package resolution - simply have the package manager resolve the packages, generate the scripts and load the scripts
Possibility. sharing packages and package caches
A major question is where packages are cached. This is primarily a responsibility of the package manager, but becomes a serious issue for scripts because packages can't be duplicated for 100s of scripts, so some shared caching is needed.
Possibility. align with C
It is possible we should approach this in a similar way that C# repl and scripting want to approach the problem for .NET Core. There are shared concerns here and it is likely we should move forward with the same model for how assemblies and assembly versions are loaded in a scripting session.
Given that C# and F# scripting are also used for Azure Functions, it seems to me that a shared behavior here would be best. However this would mean spec'ing out that behavior all-up and perhaps building out an underlying component that F# and C# could sit atop.
This means that as things stand today the implementation of F# scripting is not "a tool on top of F#" (ala scriptcs) but actually part of FSharp.Compiler.Service.dll and thus pretty much universally supported in F# editors. The allows us to deliver scripting into a very wide range of contexts simply, efficiently and consistently, e.g. into Ionide, Azure Functions, Azure Notebooks and many online tools.
With integrated package resolution one option is that this becomes
However, this approach doesn't seem to work particularly well with the incremental addition of package specifications in a REPL session.
Challenge: Compiling scripts and architectural layering
Traditionally the tool fsc.exe has supported the ability to compile F# scripts including their references. This has induced a violation of layering in the toolchain: the FSC.EXE tool also included the logic for assembly resolution and quite a lot of logic for processing F# files as scripts. This meant that the FSC.EXE tool and FSharp.Compiler.dll became badly dependent on the MSBuild API simply to resolve assembly references in scripts.
Worse still this "leaked out" into the logical specification of the compiler itself. The compiler was now able to accept strange assembly specifications such as -r:System. FooBar,Version=3.2.10,.. on the command line and resolve them with MSBuild. In the original world of .NET MSBuild shipped as part of the .NET Framework. However when MSBuid became separated this started to cause immense pain, and even more so when ,.NET Core came about. This problem was poisonous to whole toolchains built on FSharp.Compiler.Service, and we only recently did the hard work to "extract the rotten tooth" and make MSBuild optional
Challenge: .NET Core toolchain
The .NET Core toolchain changes some things about how F# compilation is surfaced. In particular it adds a layer dotnet ... via which all tools are accessed from the command line.. This means that with .NET Core the architectural layering becomes something like this:
The question is where [PackageResolution] happens in this toolchain. But first there are other questions that need to be resolved:
Not all variations of F# running on .NET Core go through the dotnet tool. For example, clients of the FSharp.Compiler.Service.dll running on .NET Core such as the Fable compiler do not - they just embed F# parsing and checking directly via the compiler service DLL.
Fable assumes an installation of "dotnet". However will it be the case that all clients of F# scripting (editors, engines etc.) can assume an installation of "dotnet"? If I embed F# scripting in an application, do I assume "dotnet" is installed? Do I have to download reference assembly packages from nuget to do simple F# script execution?
It is not clear what the specification of the F# .NET Core scripting engine will with respect to loading "uplevel" versions of the assemblies that are used to implement the scripting tool itself. This is discussed below.
It seems that these questions need to be resolved before integrated package management can be addressed.
[ edited by @dsyme to be a more comprehensive guide to this design question ]
[ Latest implementation is here: https://github.com/Microsoft/visualfsharp/pull/4042 ]
Package references in F# scripts
There has been a long standing desire to add the ability to reference nuget packages from F# scripts. Originally this was conceived as basic fully qualified nuget references like
packages.config. Lately this has evolved into integrating "dotnet" or "paket" or "npm" package specifications and dependency management tools as part of the toolchain, or providing generic hooks to allow this.Related links
#r nugetin RoslynDesign principles
You can add a package reference to a script with a single line using a normal text editor that supports F# - no extra files (e.g. a packages.config) are needed
Package references include version constraints, and dependency resolution is performed a.la. nuget v3 and/or paket
it works "at design time" , i.e. I can open a script containing package specifications and quickly get editing and type checking against a resolved set of packages - without needing to run any of the script or any command line tools
it works universally, i.e. it works on any default install of F# editing + scripting, whether Ionide or the Visual F# Tools or web-hosted delivery of F# scripting such as Azure Functions or Azure Notebooks.
It aligns well with how package management will be dealt with in the future of the .NET toolchain
it works "the same way" on both .NET Framework/Mono and .NET Core, at least to some approximation,.
It considers the needs of F# when used as a Javascript programming language through toolchains such as Fable or WebSharper. Here NPM is a natural package manager, though there are others.
The design and implementation do not induce a dependency on any one specific package manager within the core F# toolchain (i.e. compiler/scripting/editor/Fsharp.Compiler.Service/ProjectCracker). It might be that different package managers have some specific support to make them work, but we remain open to new package managers.
The implementation does not induce bad "layering problems" in the F# toolchain implementation reminiscent of MSBuild, see below.
It works efficiently - package resolution is amortized, for example.
The default settings in editing and execution tools are sufficiently space-efficient, sharing packages between scripts if needed to achieve this.
Possibiity. "dotnet" references in scripts
The way the new dotnet core tooling loads nuget packages and their assemblies is awesome! I've been using its extensibility to build a DotnetCliTool. All the dependencies are downloaded & loaded from the single
%userprofile%\.nuget\packagesdirectory. I would like to be able to use this same mechanism from scripts. I would like the same types of reference to be supported, package references and project references.Instead of having to use a nuget client to download the package and then reference the assembly like:
I want to be able to do:
Possibiity.
#project "my.fsproj"references in scriptsPeople have suggested that project references should work the same way as package references to work in Visual Studio 2017 with the dotnet core tooling. e.g. be able to do:
Possibiity.
#r "paket: Foo.Bar.dll"references in scriptsThe experiment https://github.com/Microsoft/visualfsharp/pull/2483 contains a prototype of integrating paket package management directly into the F# programming model for .NET Framework programming. This includes design-time support. The experiment violates one of the design principles above - it "bakes in" support for Paket only. However that support could be factored into either fsi.exe.
Possibility.
#r "packages"references in scriptsPeople have suggested that a script simply referencing
#r "packages"should implicitly pick up the packages from its context, e.g. the containingpackages.configorsolutionorpaket.dependenciesin the toolchain. In the Experiment adding Paket support to FSI this is#r "paket"Possibility. implicit generation of load scripts
Paket has a feature to accurately generate
.fsxand.csxload scripts containing the#rreferences suitable for use with F# and C# scripting. This feature is a natural and simple way to integrate package resolution - simply have the package manager resolve the packages, generate the scripts and load the scriptsPossibility. sharing packages and package caches
A major question is where packages are cached. This is primarily a responsibility of the package manager, but becomes a serious issue for scripts because packages can't be duplicated for 100s of scripts, so some shared caching is needed.
Possibility. align with C
It is possible we should approach this in a similar way that C# repl and scripting want to approach the problem for .NET Core. There are shared concerns here and it is likely we should move forward with the same model for how assemblies and assembly versions are loaded in a scripting session.
Given that C# and F# scripting are also used for Azure Functions, it seems to me that a shared behavior here would be best. However this would mean spec'ing out that behavior all-up and perhaps building out an underlying component that F# and C# could sit atop.
ScriptCS also wants to align with a compatible mechanism: https://github.com/fsharp/fslang-suggestions/issues/542#issuecomment-282497990
Possibility. allow expression of SemVer version constraints
Paket and dotnet both have ways of specifying version constraints. The abiility to include these prior to package resolution is important.
https://github.com/fsharp/fslang-suggestions/issues/542#issuecomment-282498554
Possibility. autocomplete and search
Auto-completing package names gives a great way to search and discover package functionality.
Basic autocomplete is possible already in package specifications like
packages.configandpaket.dependencies. e.g. see autocomplete in IonideAdditionally auto-completing on search terms such as
#r "package: statisticsgiving search of package description text would be helpful.Challenges
Some things make this tricky for F#
Challenge: Compiler architectural layering (basics)
The basic "natural" layering of the toolchain is
Here
[AssemblyResolution]is a plugin-point or the mechanism used to resolve assemblies.Note that in this architecture the F# editing tools support the F# scripting programming model via
FSharp.Compiler.Service.dll.This means that as things stand today the implementation of F# scripting is not "a tool on top of F#" (ala scriptcs) but actually part of
FSharp.Compiler.Service.dlland thus pretty much universally supported in F# editors. The allows us to deliver scripting into a very wide range of contexts simply, efficiently and consistently, e.g. into Ionide, Azure Functions, Azure Notebooks and many online tools.With integrated package resolution one option is that this becomes
where
[PackageResolution]indicates a potential plug-in point. An alternatives would be to build a layer "outside and on top" of FSI.EXEHowever, this approach doesn't seem to work particularly well with the incremental addition of package specifications in a REPL session.
Challenge: Compiling scripts and architectural layering
Traditionally the tool
fsc.exehas supported the ability to compile F# scripts including their references. This has induced a violation of layering in the toolchain: the FSC.EXE tool also included the logic for assembly resolution and quite a lot of logic for processing F# files as scripts. This meant that the FSC.EXE tool and FSharp.Compiler.dll became badly dependent on the MSBuild API simply to resolve assembly references in scripts.Worse still this "leaked out" into the logical specification of the compiler itself. The compiler was now able to accept strange assembly specifications such as
-r:System. FooBar,Version=3.2.10,..on the command line and resolve them with MSBuild. In the original world of .NET MSBuild shipped as part of the .NET Framework. However when MSBuid became separated this started to cause immense pain, and even more so when ,.NET Core came about. This problem was poisonous to whole toolchains built on FSharp.Compiler.Service, and we only recently did the hard work to "extract the rotten tooth" and make MSBuild optionalChallenge: .NET Core toolchain
The .NET Core toolchain changes some things about how F# compilation is surfaced. In particular it adds a layer
dotnet ...via which all tools are accessed from the command line.. This means that with .NET Core the architectural layering becomes something like this:The question is where [PackageResolution] happens in this toolchain. But first there are other questions that need to be resolved:
Not all variations of F# running on .NET Core go through the
dotnettool. For example, clients of the FSharp.Compiler.Service.dll running on .NET Core such as the Fable compiler do not - they just embed F# parsing and checking directly via the compiler service DLL.Fable assumes an installation of "dotnet". However will it be the case that all clients of F# scripting (editors, engines etc.) can assume an installation of "dotnet"? If I embed F# scripting in an application, do I assume "dotnet" is installed? Do I have to download reference assembly packages from nuget to do simple F# script execution?
It is not clear what the specification of the F# .NET Core scripting engine will with respect to loading "uplevel" versions of the assemblies that are used to implement the scripting tool itself. This is discussed below.
It seems that these questions need to be resolved before integrated package management can be addressed.