Support a Literate .fsx File

[jbeeko]

Is your feature request related to a problem? Please describe. When writing documentation for libraries there are two (at least) problems:

1. keeping samples aligned with the current state of the library
2. providing very low friction ways for people to try out a library

Describe the solution you'd like Fornax should support loader/generator that are able to consume .fsx files written using FSharp.Formatting .fsx literate programming pidgin. The reason this is useful now is that the new #r directive will allow those files to contain code the exercise a library with out any explicit project. Just references to Nuget.

The doc editing experience would be as follows:

• Write the sample code needed in an 'fsx file.
• Add documentation using the embedded markdown features NOTE: unfortunately there is no simple preview so that will depend on doing a build.

The docs generation pipeline would be as follows:

• Recognize .fsx file in the content area
• Apply the FSharp.Formatting transformations to HTML

The output on the site would be:

• Interleaved documentation and code describing how to use the library in a logical executable sequence top to bottom.
• Download link that let you download the source .fsx file

User interaction would be:

• Review the docs on line
• Download the .fsx and open the file in Ionide
• Execute parts of the file from top to bottom. If samples are written assuming the new 5.0 #r directive things should just work.

Why this is a Killer Feature for Fornax

1. Other static site generators can't easily integrate generation of code from .fsx files so will not be able to support the seamless read, then download, then try interaction model proposed here.
2. This approach can ensure the samples compile as the library evolves, no more outdated samples. Or samples with typo's

[Krzysztof-Cieslak]

Well, yes Fornax can support such loader - actually we’re already using FSharp.Formatting in Waypoint doc template (and in Saturn documentation) to generate API reference for libraries.

Adding loader to support literate programming sounds like a really good idea - it should be really useful for writing walk-through types of documentation, but I guess it’s an issue more for Waypoint rather than Fornax directly.

[Krzysztof-Cieslak]

Issue transferred to the Waypoint repository

[jbeeko]

My original thought was to add a loader to the Fornax project that could interpret .fsx files in content directories as literate scripts. Kind of a general purpose thing.

But since you moved this here I take it that you plan to keep the Fornax project quite minimal and have other projects create the loaders they need?

[Krzysztof-Cieslak]

Fornax is just an engine, it doesn't really make any assumptions what loaders users should have - loaders really depends on particular project needs. The blog template that we have currently in Fornax is fairly minimal and is more a sample/starting point.

However, I do have plan to add documentation template to Fornax (so you could do either fornax new blog or fornax new docs but it's not there yet, and when I'll be adding it, it will be definitely based on the documentation template we have here

The other good idea may be creating some kind of separate repository that would work as a gallery of loaders that people could just copy-paste to their projects and then personalize as needed.

[jbeeko]

he other good idea may be creating some kind of separate repository that would work as a gallery of loaders that people could just copy-paste to their projects and then personalize as needed.

I was thinking that. And what will make that a lot nicer is if loaders can be written as a pure .fsx file with refs to dll's via

r# "nuget: Fornax"
r# "nuget: Markdig"
etc

I think this would mean the entire doc site could get a lot simpler because it will not need to have the Fornax dll's. Or at least writers of Loaders will not need to know about them.

I raised https://github.com/ionide/Fornax/issues/70 because currently this does now work, I suspect due to the more complicated package structure. I'll try the same from a small project and if the ref works there will close that and raise it over on the FSharp side instead.

[jbeeko]

So I imagine the pipe line would be a bit like this:

doc.fsx -- literate script processing-----> .fsx into resources dir
                                        |
                                         -- page loader --> doc.html with link to .fsx file

doc.fsx will have the fornax top matter as it's first markdown section but otherwise be a normal such script. See the sample below. This will generate a documentation page will a link to a file with just the fsharp code in it.

Using this approach

1. the document can be edited and tested independently in a .fsx file.
2. library users get nice formatted docs with tool tips
3. library users can download the code in a .fsx file and try it out

SAMPLE Literate script.

(**
--
layout: post
title: Some nice post title
author: @k_cieslak
published: 2020-02-19
---
*)

(**
# Storage Accounts
First you need to define a storage account 
*)

(*** include: storage-account ***)

(** 
## Deploying and deleting resource groups
To deploy to Azure do this
*)

(*** include: deploy ***)

(**
To delete the resource group do this
*)

(*** include: delete ***)

#r "nuget: Farmer"
open Farmer
open Farmer.Builders

//Define a storage account
(*** define: storage-account ***)
let myStorageAccount = storageAccount {
    name "yourfirststorageaccount"
}

// Define the webApp
let myWebApp = webApp {
    name "yourFirstFarmerApp"
    setting "storageKey" myStorageAccount.Key
    depends_on myStorageAccount.Name
}

// Create the deployment
let deployment = arm {
    location Location.NorthEurope
    add_resource myStorageAccount
    add_resource myWebApp
}

// Actually deploy the resource group
(*** define: deploy ***)
deployment
|> Deploy.tryExecute "myResourceGroup" Deploy.NoParameters

// Delete the resource group
(*** define: delete ***)
Deploy.Az.delete "myResourceGroup"

SAMPLE .fsx script that can be downloaded.

#r "nuget: Farmer"
open Farmer
open Farmer.Builders

//Define a storage account
let myStorageAccount = storageAccount {
    name "yourfirststorageaccount"
}

// Define the webApp
let myWebApp = webApp {
    name "yourFirstFarmerApp"
    setting "storageKey" myStorageAccount.Key
    depends_on myStorageAccount.Name
}

// Create the deployment
let deployment = arm {
    location Location.NorthEurope
    add_resource myStorageAccount
    add_resource myWebApp
}

// Actually deploy the resource group
deployment
|> Deploy.tryExecute "myResourceGroup" Deploy.NoParameters

[Krzysztof-Cieslak]

I'm not sure if there needs to be any difference between the literate script and sample script that can be downloaded - in general, the whole concept of literate programming is about having a single source of code and documentation, so not sure why would we separate it here? The literate script will still be normally runnable from F# interactive

[jbeeko]

I think it is important that when a library user downloads a script to play with it is as unadorned as possible. It should look like a file containing all the snippets from the documentation page and not something with more documentation markup than code.

Now...... if there was an Ionide mode that would take the literate file and render the markdown formatted but still let you execute the fsharp code...... ;).

[jbeeko]

Anyway having the separate sample file is an enhancement, I think and important one but one that can be added later.

[Krzysztof-Cieslak]

I think it is important that when a library user downloads a script to play with it is as unadorned as possible. It should look like a file containing all the snippets from the documentation page and not something with more documentation markup than code.

I think this is possible to a degree. For example as far as I remember all the include and define part is optional - you can just mix comments and code freely. And for normal comments - I think it may be good to give users file with those comments as it means they can read it while playing with the file without context switching to the browser.

Now...... if there was an Ionide mode that would take the literate file and render the markdown formatted but still let you execute the fsharp code...... ;).

Heh, let me play with this idea.

[jbeeko]

Ok, I take you point, I've re-written that sample quick start to be more legible and it is fine. In fact looks great in iodide.

I added the link to the fsx file in the top matter which will self-document a bit.

(**
--
layout: post
title: Farmer Quickstart
author: @jbeeko
published: 2020-05-19
source: https://compositionalit.com/farmer/docs/content/sample.fsx
---
*)

(** 
# Farmer Quick-start
Building an infrastructure with Farmer consists of two steps. 
1. Defining an infrastructure using the Farmer Builder
1. Deploying that infrastructure either by generating the ARM template and using standard tools or deploying it directly from Farmer by having Farmer invoke the Azure CLI. 

## Defining an infrastructure
Farmer lets you define infrastructure elements by specifying the resources and then combining they into a deployment. *)

(** First we download and open the requisit Farmer binaries *)
#r "nuget: Farmer"
open Farmer
open Farmer.Builders

(** Then we define a storage account. *)
let myStorageAccount = storageAccount {
    name "yourfirststorageaccount"
}

(** Then a web app. The storage account key blah, blah. *)
let myWebApp = webApp {
    name "yourFirstFarmerApp"
    setting "storageKey" myStorageAccount.Key
    depends_on myStorageAccount.Name
}

(** Then those are combined into a deployment. *)
let deployment = arm {
    location Location.NorthEurope
    add_resource myStorageAccount
    add_resource myWebApp
}

(** 
## Deploying and deleting resource groups
The following will generate the ARM template,invoke the Azure CLI (collecting any required credentials) and deploy the template to the specified resource group in you subscription. *)
deployment
|> Deploy.tryExecute "myResourceGroup" Deploy.NoParameters

(** Finally if you want to extend the example this will delete the resource group letting to start over. *)
Deploy.Az.delete "myResourceGroup"

[Krzysztof-Cieslak]

Also, I believe you can also use normal style comments (i.e //) rather than the bloc one ((** ... *)) which would make this even clearer in some places (like those single-line comments)

[jbeeko]

You can use them but I don't think the // comments will be rendered into markdown. They will remain code comments. Which may be ok for somethings.

[Krzysztof-Cieslak]

Ah, yes, that maybe the case, indeed.

[jbeeko]

Ok, new plan:

1. Scripts should be written like this https://github.com/ionide/Waypoint/issues/21#issuecomment-633680402. Note the script includes the top-matter.
2. The .fsx loader will build a SiteContents as follows:
   1. Extract the top-matter markdown from the .fsx file and put it into the SiteContents
   2. Feed the remainder of the .fsx file into the FSharp.Formatting literate script processor getting back HTML
   3. Wrap the returned HTML into markdown using the markdown syntax indicating raw html.
   4. Add a link ref in MD syntax to the markdown.
   5. Put markdown into the SiteContents contents
   6. PageGeneration from the SiteContents can then proceed in the normal way.

This is the simplest way I can think of to enable this feature. A couple of more sophisticated alternatives:

1. hack https://github.com/jbeeko/FSharp.Formatting/blob/d8be726f4b8109d2fe95bb9a285c723f87ef599c/src/FSharp.Literate/ParseScript.fs#L12-L15 to be able to get a collection of markdown and HTML, then manipulate that to build the MD for the SiteContents
2. Rather than insert the download link into markdown create a new page type that takes a URL parameter and the display of the download link can't then be controlled by the template.