Fsc can manipulate AST at compile time (wanna approve)

[kekyo]

Fsc can manipulate AST at compile time (wanna approve)

Overview

I propose "the AST translator" is usable and easier metaprogramming at F# world.

It's handle manipulation for untyped AST at compile time, these AST nodes are: ParsedInput, SynExpr, SynPat... etc.

For example, a basic F# code fragment:

let addFunc a b =
    a + b
let subFunc a b =
    a - b

[<EntryPoint>]
let main argv =
    let r1 = addFunc 1 2
    let r2 = subFunc 1 2
    printfn "addFunc=%d, subFunc=%d" r1 r2
    0

We will be able to compile by fsc with the AST translator. Uses sample "FunctionLoggingTranslator" translator (complete code, following link)

type FunctionLoggingTranslatorImpl() =
    interface ITranslator<ICompilerConfig, ErrorLogger, ParsedInput> with
        member __.Name = "FunctionLoggingTranslator"
        member __.Translate config errorLogger input =
            match input with
            | ParsedInput.ImplFile(ParsedImplFileInput(fileName, isScript, qualifiedNameOfFile, scopedPragmas, hashDirectives, modules, (isLastCompiland, isExe))) ->
                ParsedInput.ImplFile(ParsedImplFileInput(fileName, isScript, qualifiedNameOfFile, scopedPragmas, hashDirectives, modules |> List.map Utilities.traverseModule, (isLastCompiland, isExe)))
            | ParsedInput.SigFile (ParsedSigFileInput(fileName, qualifiedNameOfFile, scopedPragmas, hashDirectives, modules)) ->
                input

[<assembly: Translator(typeof<FunctionLoggingTranslatorImpl>)>]
do ()

Made result same as:

let addFunc a b =
    System.Diagnostics.Debug.WriteLine("Enter function: addFunc")
    a + b
let subFunc a b =
    System.Diagnostics.Debug.WriteLine("Enter function: subFunc")
    a - b

[<EntryPoint>]
let main argv =
    System.Diagnostics.Debug.WriteLine("Enter function: main")
    let r1 = addFunc 1 2
    let r2 = subFunc 1 2
    printfn "addFunc=%d, subFunc=%d" r1 r2
    0

I forked and implemented it from visualfsharp repo. See my forked repo 'ast-translator-test' branch.

I wanna approve it and finally merge main repo. Of course, there is preparation to correct the pointed out problem.

Description

The AST translator is full-customizable for untyped AST at compilation time. Therefore we can handle additional features with no runtime costs (compare to quotation expressions and monadic structures)

We'll receive better way:

• Automated translation source code at compilation time.
  • We can handle metaprogramming approach with no additional source code maintenance cost (versus source code generative tools.)
• Type safe code fragment manipulation.
  • The untyped AST node type is purely F# union-case and/or record types.
  • These elements are safely, but AST translation is difficult (see middle ware section.)
• Simple architecture at fsc.
  • Fsc contains only calling AST translators. Translation details totally outside for fsc.
  • I carefully designed to be minimal code changes.
• Easy understanding, better compatibility architecture.
  • If we don't use AST translator, fsc is totally full-compatible, nothing any changes. We can understand it has no effect.

We'll be able to use for:

• Generative programming at compilation time.
• Aspect orientation programming at compilation time.
• Pluggable metaprogramming basis architecture.
• And more...

Details

How to use this way

We can easy to use AST translators:

• We have to make simple class type inherit from ITranslator<ICompilerConfig, ErrorLogger, ParsedInput> and implements the Translate function.
  • Signature: Translate ICompilerConfig -> ErrorLogger -> ParsedInput -> ParsedInput
• Use fsc with --translator:<path> argument.
  • It can give multiple, means we can apply multiple AST translators at a compilation time.

Overall architecture

The AST translator uses F# untyped AST node types at FSharp.Compiler.Private.

1. Fsc parses and aggregates command line option with --translator:<file>. Load these translator assemblies.
2. Fsc finds translator by the indicator for TranslatorAttribute attribute type applied at translator assembly.
3. Instantiate translator type from attribute information (TranslatorAttribute.TargetType.)
4. Call to translator type's Translate function.
5. The translator can anything AST manipulation and return new AST. The AST node is ParsedInput.
6. If avaliable another translators, repeats between 4 and 5.
7. Fsc continues for compilation with translated AST.

     +---------------+   * TranslatorAttribute
     |  Fsharp.Core  |   * ITranslator<_, _, _>
     +-------+-------+
             ^
             |
+------------+--------------+   * ICompilerConfig (derived to TcConfig)
|  FSharp.Compiler.Private  |   * ErrorLogger
+--------+-------------+----+   * ParsedInput (etc...)
         ^             ^
         |             |      +-----------------------------+
         |             +------+  FunctionLoggingTranslator  |
         |                    +-----------------------------+
         |                                 ^  __.Translate ... -> parsedInput
   +-----+-----+    Finding and calling    |
   |  fsc.exe  | <-------------------------+
   +-----------+

Featuring middle ware

(The concept not evaluate any code fragments, but I'm clear for this idea)

We can use the AST translator now, but untyped ASTs manipulates difficulty for common users. I'm expecting to develop middle ware library by communities.

The middle ware is fully AST translator but it requires additional translation information from source code. To give an example, we wanna insert arbitrary enter-exit code fragment:

[<InsertBefore>]
let beforeFunc (name: string) (args: obj[]) : unit =
    printfn "Enter function: %s %A" name args

[<InsertAfter>]
let afterFunc (name: string) (args: obj[]) (result: 'a) : 'a =
    printfn "Exit function: %s %A, Result=%A" name args result
    result

// ----------------

let addFunc a b =
    a + b
let subFunc a b =
    a - b

[<EntryPoint>]
let main argv =
    let r1 = addFunc 1 2
    let r2 = subFunc 1 2
    printfn "addFunc=%d, subFunc=%d" r1 r2
    0

The (imaginary) middle ware AST translator find InsertBeforeAttribute and InsertAfterAttribute annotated functions and insert debug output to use these functions. We can write it translator (exactly, but difficult) and it'll make results:

[<InsertBefore>]
let beforeFunc (name: string) (args: obj[]) : unit =
    printfn "Enter function: %s %A" name args

[<InsertAfter>]
let afterFunc (name: string) (args: obj[]) (result: 'a) : 'a =
    printfn "Exit function: %s %A, Result=%A" name args result
    result

// ----------------

let addFunc a b =
    let __args = [|a;b|]
    beforeFunc "addFunc" __args
    afterFunc "addFunc" __args (a + b)
let subFunc a b =
    let __args = [|a;b|]
    beforeFunc "subFunc" __args
    afterFunc "subFunc" __args (a - b)

[<EntryPoint>]
let main argv =
    let __args = [|argv|]
    beforeFunc "main" __args
    let r1 = addFunc 1 2
    let r2 = subFunc 1 2
    printfn "addFunc=%d, subFunc=%d" r1 r2
    afterFunc "main" __args 0

The middle ware translator gets additional translation information via:

• From source code (above)
  • The attribute type, fixed formal signatured functions, etc...
  • If translator requires additional types/functions (including attribute for example), we need to bundle with compile-time and/or runtime libraries. It's bit difficult at the (nuget) package structure.
• From additional translator directive file
  • AST top node contains where's source code path on the file system.
  • We can refer source code path and read additional files from same directory.
• From compiler configuration
  • ICompilerConfig interface gives information for the fsc commandline options.

AST translator assembly loads way

The AST translator assembly will load by fsc's command line option --translator:<file>.

• Basic operation: total manually usage for fsc at command line.
• Uses by MSBuild: FSharp.Build can manage the translator option. It aggregates from targets script in ITaskItem[] symboled TranslatorPath.
  • Many users have to add TranslatorPath element into theirs *.fsproj manually. (Implemented but not tested)
  • Or, if we make nuget package for translator middle ware, it can include custom *.targets MSBuild script and declares TranslatorPath element for auto configuration. It means most users only append the nuget package into thier project and done!

TODOs

• Improves translator if support both untyped AST and "typed AST."
  • The interface ITranslator<...> has capable it senario. We can change third type arg ParsedInput to typed ASTs.
  • The translator requires more additional configuration related informations/functions. Because typed AST maybe requires assembly/type resolvers.
  • (It's difficult for me, I don't understand typed ASTs now. So I'll do next step if this feature needs.)
• The AST translator has to add reference to FSharp.Compiler.Private.
  • Because a lot of untyped AST node types declare at this assembly.
  • We know these types declaring at FSharp.Compiler.Service too.
  • Unfortunately, currently can't use translator with FCS.
  • I wanna move commonly AST node types into FSharp.Core or FSharp.Compiler.Core named like new interface assembly. Structure below:

     +---------------+                +---------------------------+
     |  Fsharp.Core  |                |  FSharp.Compiler.Service  |
     +-------+-------+                +----------+----------------+
             ^                                   |
             |                                   |
+------------+-----------+  AST node types       |
|  FSharp.Compiler.Core  +<---------+------------+
+------------+-----------+          |
             ^                      |
             |                      |
+------------+--------------+       |
|  FSharp.Compiler.Private  |       |
+--------+-------------+----+       |
         ^             ^            |
         |             |      +-----+-----------------------+
         |             +------+  FunctionLoggingTranslator  |
         |                    +-----------------------------+
         |
   +-----+-----+
   |  fsc.exe  |
   +-----------+

Background

First way

The way was beginning from runtime validation logger using quotation expression (backend by FSharp.Quatation.Compiler)

• We used quotation expression infrastructure because we made the middle-ware framework library. Customers needed separation of concern between implementation detail and validation logger.

But it has a problem for very slow at first runtime execution (cause internal compilation process), we can't pay it cost for business.

Second way

We're starting the fscx project. It's first approach for AST manipulation mechanizm using FSharp.Compiler.Service.

It project mades better results for the purpose. But it had these problems:

• fscx compilation backend is used for FCS. It means fscx didn't have full compatible for fsc (native F# compiler.)
• Too complex architecture. Because we have to safely replace from fsc compiler to fscx compiler on MSBuild/NuGet infrastructures.
  • With assembly loader problem same as F# type provider's difficulty if packaged.

Now

Finally, fscx project was suspended (for only business reason.) I wanna finish it and I was thinking about what's more appropriate in what form.

I reimplement from scratch in the F#'s repo direct without FCS.

Extra information

Estimated cost (XS, S, M, L, XL, XXL): M

Related suggestions:

• The fscx project
• About Expandable F# Compiler project (blog entry)

Affidavit (please submit!)

Please tick this by placing a cross in the box:

• [x] This is not a question (e.g. like one you might ask on stackoverflow) and I have searched stackoverflow for discussions of this issue
• [x] I have searched both open and closed suggestions on this site and believe this is not a duplicate
• [x] This is not something which has obviously "already been decided" in previous versions of F#. If you're questioning a fundamental design decision that has obviously already been taken (e.g. "Make F# untyped") then please don't submit it.

Please tick all that apply:

• [x] This is not a breaking change to the F# language design
• [x] I or my company would be willing to help implement and/or test this

[robkuz]

Would it be possible to manipulate also types with this approach?

[kekyo]

@robkuz Sure.

We can manipulate all F#'s type declaration.

• Add, remove and modify all members.
• Add, remove, and modify all attributes.
• And can modify member bodies/function expressions.

For example:

• We can change from declared record type to class type.
• We can auto implements with IPropertyChanged interface.
• We can auto implements for generative serialization code fragments (achieve no runtime cost.)

[7sharp9]

I don't think this is flexible enough for a general plugin, what I would want is to transform the AST and splice into type holes, or transform sections of annotated code entirely. Then again maybe I misunderstood the example, maybe a more through example would enlighten me.

[kekyo]

@7sharp9 Hi, thanks reply. Do you need more example for understanding, or need a example for deeper? (Please you should say me if I mistake understand it :)

I'll dig overview a little. I think about what the translator good at:

• Easy to use translation: AST translation detail is not easy (I understand). But constructed middle ware translator usage is very easy (maybe only load by NuGet). I hope communities publish translator for common usage. It likely the type providers.
• We will use the translator when it is effective for better (or NO) runtime cost.

I can tell it imagine very narrow situation for use. But I feel the translator will be a lot of application possibility. These examples below can fix by the translator. And maybe we make the middle ware for it.

---

IPropertyChanged auto implementer.

It interface implementation are a lot of methods invented.

• Fully manual implements. Official example
  • We have to implement repeatedly too tired...
• Stack overflow entry What are a lot of answers!! (I surprised...)
  • I experienced my few team members are often mistakes lack or invalid for invoking Changed() method (or boilerplate method).
• How to pick up the property name:
  • C#: nameof expression, Use LINQ expression and extract from it.
  • F#: nameof expression, Use quoted expression and extract from it
  • The expression method have runtime cost.
• Use ReactiveProperty It's excellent library but:
  • The view side binding expression has to add ".Value" property reference. If it lacks, will fail binding silently...

The serializer traversal handler.

• Many implementation (Json, Xml, Binary etc...) traverse type structure using reflection. It's very slow.
• Faster implementation uses for:
  • Use LINQ Expression internal compiler. It's better way but slower at first execution and cannot (or very limitation) use on the mobile platform. Or our F#, we can use FSharp.Quotation.Compiler but has same problem.
  • Manual implements custom traverser... We have to maintanance these (non related to business) fragments.

---

The translator can fix for general problems, but it's invalid way. I feel it uses:

• Cannot fix by standard F# code fragments/functional or OO structures includes type provider, computation expression and quotation expression.
• (I don't wanna use this word for misunderstanding...) If we have to insert for aspect fragment entirely and/or marked position likely AOP.
• Dislike or totally impossible the runtime cost. (And cannot use runtime infrastructure)

[dsyme]

This is macros of SyntaxTrees by plugins. It is not something we plan to add to F#, see #210 for example and all the discussions there