I think the ProjectScaffold follows a pattern usual in all F# projects - it started with two fairly simple script files (generating documentation and FAKE build script) that keep growing and getting more complex. I think this is quite natural (Literate documentation in F# Formatting started as a script file too :-)). But I think that ProjectScaffold has now reached the level when it is too complex to be just two script files.

At the moment, there is quite a lot of logic in the two script files that a person using ProjectScaffold for the first time should not see and should not care about (say, the workaround to make Razor work on Mono). Don't get me wrong - all this logic is really useful and it's great to have it - but we should not "bother" the first-time user with it.

Also, there is quite a lot of variations on what kinds of projects people may want to create with ProjectScaffold (multi-project vs. single-project, type-provider vs. normal, multi-language, etc.).

So, I think we should extract some of the functionality into a library & simplify the template. Here is what I think we should do:

We should create a library (it could be called ProjectScaffold.dll or something better) that references F# Formatting and FAKE and contains some of the functionality from both build script & docs generation.
I like the structure that FsReveal uses - we can have master branch which contains the template (which relies on ProjectScaffold.dll and gets it from NuGet via Paket). Then we can have other branch (say, develop or library) with the more complex functionality.
This would let us keep the master branch relatively stable and it would make it easier to update projects to latest version of ProjectScaffold (just paket update). The ProjectScaffold NuGet package can contains fsx files that we #load from the template (this way, it is easier to add new dependencies if we need to, or change paths etc.).
Because master would be simple, it would also let us create multiple different templates. (Because most of the work would be done in the library).

The only other alternative I can think of would be to move some of the ProjectScaffold functionality into F# Formatting and FAKE. But I don't like this that much - F# Formatting and FAKE should not be tightly linked with ProjectScaffold.

Also, I think we could consider eliminating separate generate.fsx script for the documentation. If the ProjectScaffold library is sufficiently easy, we can just have documentation generation as 3 targets in the build script. Finally, I think that the best way to design the ProjectScaffold library API is to use something like Scaffold.GenerateReference(projectInfo, ... ) with projectInfo being some record or a class (wrapping shared info about project) and ... are optional parameters.

Now, this changes project scaffold from being just "scaffolding" into another stand-alone component that people can use. But I think this is inevitable at this point - because it really is not a "scaffolding" project anymore (in the sense that you cannot just clone it and modify bits of it - even I don't understand everything that the template does these days!)

Another useful project that we should look at is https://github.com/matthid/Yaaf.AdvancedBuilding by @matthid, which he mentioned here: https://github.com/tpetricek/FSharp.Formatting/pull/275

I personally don't really like the ProjectScaffold layout (I think the layout is unnecessarily complicated). Normally I use my own (https://github.com/matthid/Yaaf.AdvancedBuilding) which updates with paket update , so I'm not always up2date with ProjectScaffold.

I think that what I'm suggesting here is actually closer to what @matthid is doing. And I 100% agree about the complexity of ProjectScaffold.

Well, initially AdvancedBuilding was a project to improve support for building for multiple targets, but it didn't worked out the way I imagined. As a matter of fact msbuild is good enough if you work with the platform variable (which seems to be never used besides AnyCPU, and actually works on recent xbuild/mono as well). As a matter of fact, if used in a specific way you can get pretty good multiple platform support with F#/FAKE But you need to follow some strict guidelines like https://github.com/matthid/Yaaf.Logging/blob/develop/src/source/Yaaf.Logging/Yaaf.Logging.fsproj#L54 has to match your configuration in buildConfig.fsx, see below. So, you should completely ignore the source code in AdvancedBuilding, it is basically only used for testing the generic build script. I played with the idea of automatically converting existing files to the better format but it's usually a good idea to know your msbuild files and edit them manually.

The general idea is this:

Have some very simple `build.sh and build.cmd which basically call into packages/Yaaf.AdvancedBuilding/content/build.sh/.cmd these files then prepare the environment and call paket or nuget to restore all build dependencies. Once everything is prepared for the build procedure they call FAKE build.fsx as usual. Note that now build.sh and build.cmd can be updated via paket update.
Like the build scripts you can opt-in into an update-able FAKE script as well, however this is a bit more difficult:
- First include packages/Yaaf.AdvancedBuilding/content/buildConfigDef.fsx To define your build configuration (basically a simple record definition)
- Then you load an instance of this record by including buildConfig.fsx (you could define it in build.fsx as well but I rather enjoy splitting the configuration from the actual build code)
- Then you simply include packages/Yaaf.AdvancedBuilding/content/buildInclude.fsx which predefines generic FAKE targets while using your (above loaded) configuration.
- Note how you can define additional configuration in buildConfig.fsx and additional FAKE targets/dependencies in build.fsx.

Some things I specifically don't like about ProjectScaffold (for this issue):

You cannot update with it (once cloned you are on your own)
the docs folder: Whenever I go in there I need to search through all folders to find what I need.

In AdvancedBuilding I use this: generateDocs.fsx is in root directory, the *.md, *.fsx files are directly in /doc or any subdirectory, then I have doc/content which is recursively copied to the dest (for content like pictures/css/js) and doc/template for all templates. That's it.
The simple /bin folder: When you opt-in to AdvancedBuilding you will get a per platform output directory like /build/net40 build/net45 (this is configurable). I think even when you only use a single platform/target this helps you by early discovering possible problems when you for example upgrade from net40 to net45.
docs/output: As we are now able to build the website multiple times (almost for free) I would suggest to always build local documentation as well. In AdvancedBuilding I build the documentation to release/documentation/local/html and release/documentation/github/html. Additionally I build the latex templates (because why not). It's actually nice to quickly open the local documentation and check everything before uploading to github.
release: actually I copy everything to release at the end, so the created nuget package is in release/nuget, you can find the relevant .dll files in release/lib/{platform} (without any dependencies like in build/{platform}) and the docs (see last paragraph).

From all the above the docs directory is definitely the biggest pain. You could argue that my solution has potential clashes. But actually because only *.md and *.fsx files are processed and templates only are *.cshtml there is not much that can happen. So the only clash that can happen is if you want http://website/content/file.html (ie a file.fsx in the content folder), however this case doesn't really hurt as we would have file.fsx and file.html in outdir/content/

So we could do something similar for ProjectScaffold to make it more "update-able". We could create some generic re-usable script-files and reference them via paket, however I don't think (from my experience with AdvancedBuilding) we can make ProjectScaffold a library (or at least I couldn't do it).

Also, I think we could consider eliminating separate generate.fsx script for the documentation.

While I agree in theory, it is sometimes nice to have a standalone documentation generation. While I see that a extra FAKE target "should" be handling this as well, I never actually managed to make the --single-target feature always work (especially taking mono, build.cmd and build.sh into account). I actually use https://github.com/matthid/Yaaf.AdvancedBuilding/blob/develop/packages/Yaaf.AdvancedBuilding/content/buildInclude.fsx#L131 (like even before FAKE had the single-target switch) and it works perfectly (you can even define extra dependencies for the _single version). The reason why I didn't combine generate.fsx and build.fsx in AdvancedBuilding is actually a bootstrapping thing in RazorEngine (it just isn't possible there). Also when you have multiple build dependencies depending on different RazorEngine.dll version you will run into troubles. By separating the scripts you don't risk running into that (I don't think anybody actually has such a setup, besides RazorEngine itself).

TL;DR

Maybe one (the simplest) solution to reduce the ProjectScaffold complexity is to use AdvancedBuilding (Note that you do not actually need to change your current layout here at ProjectScaffold as most folder-paths are actually configurable in AdvancedBuilding, the defaults are just different :) ). This way you get back to very, very simple build.fsx, build.sh, buildConfig.fsx generateDocs.fsx and build.cmd. And as long users don't change any defaults their build scripts will be updated with paket update.

My main concern if moving to a library type project is that we don't destroy the ease of customizing.

IE: Scaffold is (at least for me) a great starting point, but I need to tweak it a bit to do exactly what I want. Currently It is relatively easy for me to maintain my project specific tweaks while keeping up to date with the latest and greatest.

As long as we simply move helper functions to a library in order to clean up build.fsx then we are probably good.

Another concern is that not being just a script will grow ProjectScaffold into a pseudo FAKE thing. IE: If your script gets too big, there is quite a big motivation to move helpers into libraries where they belong: Like extending Paket, or FAKE, or maybe even creating some specific library for a specific feature. When ProjectScaffold becomes a library, it will no longer feel messy to add a bunch of helpers there, And it will be easy to end up with a lot of paralell code that could perhaps be generalized and put into FAKE or other projects.

What to do?

My suggestion would be to extract helpers and functions into a FAKE sub project, (avoiding hard dependencies from FAKE to all the other projects scaffold users), but keep them in the FAKE github repo. This way we will keep a single repo of FAKE related build helpers avoiding duplicate efforts. And it is easier for maintainers to have a good overview and make decisions on what features belongs where.

It would be really nice to have generate.fsx as a library that can be parameterized in a nice way, as it stands the errors I've gotten when things break with it are awful and so we've disabled it in most of our projects. I think most people only need the standard use case and customization could always be done by forking and building it yourself.

I like the idea of having it as a library too. I don't think we should be relying on git or creating complicated guidelines and file layout patterns to be able to 'fork' a project out of scaffold and then keep it up to date. At the moment agree that keeping up to date is too hard. I typically don't, until things break, and then it's quite an annoying job to come here and figure out what I need to change. Also if I want to use a new feature it's too onerous to backport it.

Maybe clojure's leiningen is a good place to look for inspiration. In particular, eventually I think this should evolve in a plugin-based architecture, where plugins can mainly contribute new project templates, fake targets and paket dependencies. Then project maintainers could write their own plugins for scaffold, and the whole thing would become less monolithic and imposing.

Several of us discussed this topic yesterday on Slack and proposed using Yeoman generators. Yeoman runs on node, but the upside is that it exists, provides a cross-platform story, and follows the approach taken by the ASP.NET team. If nothing else, it could provide a path to something better.

Following Ryan comment - discussion took place on F# slack channel (http://fpchat.com/), I'd love to invite anyone who still is not hanging out there ;)

[To be clear: If I mention project I mean fsproj level, ProjectScaffold is solution level tool] And to the topic (and maybe bit off-topic). I really think we should try to provide good way of scaffolding different type F# projects not depending on VS and VS Gallery. While ProjectScaffold is doing decent / good job in providing general structure and set of commonly used tools for whole solution set up, I feel the biggest "point of pain" right now is creating F# projects for different types of applications.

[To be clear again: I have no idea how complex are those tasks as such, I've never written Yeoman generator] Generator approach( Yeoman was suggested since it's most popular and pushed by ASP .Net team tool) would allow us to create better user experience in creating F# solutions (for example asking user what tools to add etc.) but also It could be base to provide good project level scaffolding. Users could be asked what is his project target, what .Net / F# version, what type of projects too add (MVC, WebAPI, WPF, Console, Suave, whatever we would want to include).

While all those things could be done in some script files "by hand" I suggest using existing widely used infrastructure. Yeoman is popular tool, it's multiplatform, it's going to be popularized in .Net world by ASP.Net team ( we may be saying: "Yes, we also have it! Just come and try F# out" ;) ), there exist integrations of yeoman with text editors (Atom sample - https://atom.io/packages/atom-yeoman). From what I understand yeoman generators are written in JS... but we have some experience with dealing with this unpleasant fact ;)

And my last point (I guess it would be not trivial to implement). For project level scaffolding I would love to see (some time in the future) system similar to Atom Package Manager - Packages ( in our case project templates) are just github repositories, Atom package "server" just registers address of repository ( + some rating / download number) and that's all. It would be cool if people could just publish templates on GitHub and we would be able to use them.

Yet again: this post is result of some my ideas mixed with ideas of other guys from Slack channel (@panesofglass, @pblasucci, @cumpsd). I have no idea how complex it would be from point of view of ProjectScaffold (I've never even contributed here, just a "from-time-to-time" user) nor how complex it is from yeoman point of view ( but I know @cumpsd has some experience in that field)

Well, I think that's all :)

2 points:

Big picture

I strongly agree that F# should provide its own development environment story. We shouldn't depend on MS doing it for us, especially not cross platform. You could argue that MS is making the move towards OSS, which is like a huge container ship making a slow turn. However, over time there are always issues popping up with something that appears to be internal MS. (OWIN story, weird handling of GH issues, reinventing everything every x time, creating issues to announce things and saying they don't actually want feedback)

Can we look at MS and hook onto things they do? Sure. Should we depend on them to make our story? No.

Right now, the F# story exists out of a very limited set of tools, you can use Visual Studio to get everything, or Xamarin. Not sure how far you can get with vim at the moment, but things like Sublime, Atom, we have bits, but no full story.

If I'm wrong, I would be more than happy to be corrected :) If someone has a way right now to do a complete F# project, without touching VS, let me know :)

Scaffolding

As for the yeoman story. I do believe it is a good tool, it fits in the command line view most F# people have, together with editors like Atom.

We have to think on what we want to scaffold. Do we want to scaffold project templates? Do we want to scaffold certain file skeletons?

I guess the easiest would be to model it like the aspnet generators, where they do indeed clone GH repos which serve as templates.

If we can discuss a bit more on it, I could have a look at starting up the yeoman story.

In my team we maintain a lot of projects, they are all C# and need to support an ancient .NET runtime (3.5), but were all created with ProjectScaffold anyway.

Obvious problems:

No way to upgrade/update (build.fsx/generate-scripts/dependencies)
Syncing of multiple projects is therefore impossible

We introduced a base build-script that we download as a paket http dependency so we can add stuff across multiple projects. In a projects we ended up just using a "settings.fsx" that provides the project config for the build and can override existing tasks:

namespace Wooga

#r @"packages/FAKE/tools/FakeLib.dll"

open Fake

module Settings =

    let project = "Wooga.SomeProject"

    let shortIdentifier = "someProject"

    let summary = "Some project"

    let description = "Some Project"

    let authors = [ "Some authors" ]

    let tags = "some"

    let solutionFile  = "Some.Project.sln"

    let testAssemblies = "tests/**/bin/Release/*Tests*.dll"

    let gitOwner = "some" 

    let gitHome = "https://github.com/" + gitOwner

    let gitName = "Some.Project"

module Build =

    Target "ReleaseDocs" (fun _ -> ())

[...]

A good starting point to improve things would be, instead of having these values replaced inside a template, we should put them inside a config file (json/yaml/xml)...

@devboy You just described in a nutshell what Yaaf.AdvancedBuilding is doing, the difference is that the "config" itself is in F# (https://github.com/matthid/Yaaf.AdvancedBuilding/blob/develop/buildConfig.fsx) instead of json/yaml/xml. It's just that it is currently trimmed to my use-case (using nunit for example and no SourceLink support to be able to build on linux -> maybe portable pdb allow this in the future). But it is easily extendable.

FAKE could be the model to emulate for the evolution of project scaffold. It could be run as a commandline tool, or setup with a fsx. The core library handles dealing with templates, the filesystem, fsprojs, solutions, etc. Specialized kinds of projects (MVC, WebAPI, WPF, Console, Suave) get their own modules with helper functions for their setup & configuration. It can ship with a set of config scripts for supported project types (scaffold templates for all effective purposes), with the standard ProjectScaffold layout and packages for F# OSS development as the default.

It could add a scaffolding DSL with helpful operators for setting up the folder hierarchy like

  Dir"project_root"
      |- (Dir "tools"
          |- "generators"
          |- "templates")
      |-"project files"
      |-"tests"

When you want to setup your project you pass the exe a config script and a target dir (or do the same in FSI) and it constructs your scaffold at the location.

Once it's a library it'd be really easy hook it up to a posh module.

fsprojects / ProjectScaffold

Grow scaffold from "script" to "library" project #154

TL;DR

Big picture

Scaffolding