For a language made in F# that compiles to F#, how should more complex types such as tuples and records be integrated?

Sorry about posting this here, I've asked on SO twice now, but I have the feeling that what I am asking them is beyond their area of expertise and I want the people who actually have the knowledge to answer my question to see it.

Since February, I've been working on Spiral, a functional language inspired by F#, but with first class staging and intensional polymorphism. What I need is some advice of how to manage integration between my own language and .NET. I can manage primitive types just fine, but I'd like to be able to use tuple and record types in .NET Dictionaries and such. Unfortunately, I do not have a way to substitute the generic parameters with Spiral's built in types. I can generate them just fine during codegen, but cannot make them pass the language boundary to .NET.

What I have been doing so far during typechecking is converting primitive types to the equivalent System.Type using typeof, but obviously that kind of thing cannot work for types represented with discriminated unions.

In the last few days, I've had a perfectly sensible idea on the face of it, instead of trying to figure out an exact type, make a token type inheriting from System.Type that carries metadata on which Spiral type it represents and use it to perform substitution on generic .NET parameters.

There are two problems with that approach - the first is that I have to implement like 30 different abstract members for my token type. And the second is that the complete implementation of a System.Type which is System.RuntimeType has a size of 6.1k LOC. For reference, my entire compiler is 3k LOC. This is making me think that I might be on the wrong path here. It is not entirely obvious to me that scavenging through the C# implementation of System.RuntimeType is really supposed to be the move I am supposed to be making here, but that is what I am going to do unless I get some better advice here.

As an extension of my previous question, I also want to ask whether I am doing the right thing by using the methods from the Reflection namespace to handle queries to .NET types for their method types and arguments or whether there are better ways that I am not aware of?

That is all I wanted to ask.

The Spiral language is nearing its first release and is quite useful in its current form though it needs to be debugged. The only major feature which I have no idea how to implement would be the above and would be grateful for being shown how. Currently I am in the process of debugging it and making some basic libraries for it. The language strongly takes after F# in terms of syntax if not semantics, and will be hugely beneficial in making of a machine learning library after I complete the Cuda backend for it. If you are interested, take a look at the last two links to get a sense of it.

Sorry about the repo being so messy - the development of the language was outgrowth of the ML project I was doing and I never decided to move from there. I'll do so once I have enough to make the first release.

To be honest I'm not very sure what you are actually asking.

First I'd like to clarify some things: Are you sure System.Type is the correct abstraction? For your own types you are compiling, there cannot be a System.Type until compilation has finished, correct?

If you compile your language to F#, I guess you somehow need to keep track of a mapping between your own type and what you have generated? Also you talk about a language barrier, but what does that actually mean? Is your interpreted or statically compiled to code (or both)?

Sorry if I completely missed what you are asking for but maybe clarifying helps others to step in as well. Maybe you can give examples of what you are trying to achieve?

Very well, I'll try my best to explain. Let me start by pasting the the type definition for Spiral's types.

and Ty =
    | PrimT of PrimitiveType
    | VVT of Ty list
    | LitT of Value
    | FunT of EnvTy * FunType
    | FunStackT of ConsedNode<EnvTerm> * FunType
    | FunHeapT of ConsedNode<EnvTerm> * FunType
    | ClosureT of Ty * Ty
    | UnionT of Set<Ty>
    | RecT of int
    | ArrayT of ArrayType * Ty
    | DotNetTypeRuntimeT of Node<Type> 
    | DotNetTypeInstanceT of Node<Type>
    | DotNetAssemblyT of Node<System.Reflection.Assembly>

I won't go into details what all of the above does, some of it is obvious by name, but the important part that I want to highlight would be last 3 types for those are the ones I am using for .NET interop.

Now suppose I wanted to use a type from the .NET land. First I would load the assembly.

inl system = load_assembly .mscorlib

Now I have a variable system with type DotNetAssemblyT. Then to make use of the .NET type in the assembly, first I have to bind a type to a variable.

inl dictionary_type = system."System.Collections.Generic.Dictionary`2"

Now I have a variable dictionary_type of type RuntimeTypeT.

But this is not an instance of type. To call a constructor on that type and instantiate it, I would need to do something like:

inl dict = dictionary_type(int64, int64)(128i32)

The first application is for generic parameters, when I do dictionary_type(int64, int64), I also get back a type, but with the generic parameters substituted for ints (which is 64-bit by default in Spiral.) Then I call the constructor for the dictionary with the capacity set to 128 (more specifically the 32-bit signed variant of that number.)

Then I can use the instance such as so:

dict.Add(1,2) |> ignore
dict.get_Item 1

I hope this much is clear. Here is a reprint of the above for easier viewing. This is essentially a condensed version of test15 from the repo, so you can see that the code does in fact look like this in the actual language.

inl system = load_assembly .mscorlib
inl dictionary_type = system."System.Collections.Generic.Dictionary`2"
inl dict = dictionary_type(int64, int64)(128i32)
dict.Add(1,2) |> ignore
dict.get_Item 1

Let me illustrate what goes on in the compiler in that same example on the first two lines using an analogy with F# code. The following is in fact what the compiler does minus a bunch of pattern matching.

open System
let system = Reflection.Assembly.Load("mscorlib")
let dictionary_type = system.GetType("System.Collections.Generic.Dictionary`2")
let dict = dictionary_type.MakeGenericType [|typeof<int64>;typeof<int64>|].GetConstructor([|typeof<int>|])

Since I am doing type inference instead of creating types dynamically, here is where the connection with the Spiral example breaks. What happens is the DotNetTypeRuntimeT is mapped to a DotNetTypeInstanceT and the code generator in the later phase knows what to print out.

Now let me highlight a certain part in the last line, namely .MakeGenericType [|typeof<int64>;typeof<int64>|]. This represents a dictionary with a key of type int64 and a value of type int64.

Suppose I wanted to make the key a tuple of int64 * int64. In Spiral I'd write it as such:

inl dict = dictionary_type((int64, int64), int64)(128i32)

Right now if I tried this, I would get an exception because the compiler to does not know how to convert the above to the equivalent of .MakeGenericType [|typeof<int64 * int64>;typeof<int64>|]. Here is how the conversion function looks like straight from the source.

    let rec dotnet_ty_to_type (x: Ty) =
        match x with
        | PrimT BoolT -> typeof<bool>
        | PrimT Int8T -> typeof<int8>
        | PrimT Int16T -> typeof<int16>
        | PrimT Int32T -> typeof<int32>
        | PrimT Int64T -> typeof<int64>

        | PrimT UInt8T -> typeof<uint8>
        | PrimT UInt16T -> typeof<uint16>
        | PrimT UInt32T -> typeof<uint32>
        | PrimT UInt64T -> typeof<uint64>

        | PrimT Float32T -> typeof<float32>
        | PrimT Float64T -> typeof<float>
        | PrimT StringT -> typeof<string>
        | PrimT CharT -> typeof<char>
        | ArrayT(DotNetHeap,t) -> (dotnet_ty_to_type t).MakeArrayType()
        | ArrayT(DotNetReference,t) -> (dotnet_ty_to_type t).MakeByRefType() // Incorrect, but useful
        | DotNetTypeInstanceT (N x) | DotNetTypeRuntimeT (N x) -> x
        | _ -> failwithf "Type %A not supported for conversion into .NET SystemType." x

The x: Ty here is the type I pasted here at the beginning. As you can see, I can only convert primitive types to System.Type which I am using to interop in the above illustrated manner. There are also arrays and the .NET types which already exist are just fetched directly.

I don't know how to convert tuples such as VVT [PrimT Int64T; PrimT Int64T] to System.Type. Using typeof is not an option because it needs to know the type it is making at compile time. I also have types other than tuples I want to support interop with.

What I am asking here is essentially how to upgrade the above conversion function so it supports the full bevy of Spiral types. Or alternatively, I am open to a better way of doing the entire interop.

This is it for my explanation for now. If there are points that are unclear then tell me and I will elaborate them further.

Let me note that Spiral does not use the standard Hindley-Milner type inference, rather it uses abstract interpretation that would be more at home in a dynamic language, though the language itself is fully static. It walks the program for start to finish and uses join points to prevent itself from diverging. In fact, the compiler could be classified as an online partial evaluator. Type inference, or more precisely propagation is done alongside a host of other optimizations.

So is (in your example)

open System
let system = Reflection.Assembly.Load("mscorlib")
let dictionary_type = system.GetType("System.Collections.Generic.Dictionary`2")
let dict = dictionary_type.MakeGenericType [|typeof<int64>;typeof<int64>|].GetConstructor([|typeof<int>|])

The final result or only what the compiler is doing to resolve everything?

To get a type instance representing a tuple you can use the Microsoft.FSharp.Reflection namespace in particular the MakeTupleType-Function. Similar stuff is available for Union-cases and records (runtime detection and construction), though the API is quite "minimalistic".

In general reflection has quite an performance impact and there is no reason to use it (at least not as final result) as your compilation could easily emit

let dict = System.Collections.Generic.Dictionary<int64,int64>(128)
dict.Add(1,2) |> ignore

Though I might still not have understand everything ;).

For other stuff in your Ty types you might need a pre-compiled runtime-library (like FSharp.Core for F# itself) to properly resolve them to System.Type if you cannot represent them with existing structures or if you cannot easily convert them in the compiler (or in the emitted code).

The final result or only what the compiler is doing to resolve everything?

It is only what the compiler is doing to resolve everything. The System.Types are only used at compile time and never at runtime. For example, once I have a variable of type DotNetTypeInstanceT of the aforementioned dictionary and I want to call Add on it here is what happens inside the compiler.

dict .GetMethod("Add",[|typeof<int64>;typeof<int64>|])

The reflection functions are only there to let me query the .NET API and nothing else. The System.Type are being used only to establish a correspondence between .NET and Spiral and instances of them are never created as they would be in a dynamic language.

For that reason, while it is great that you pointed me to the Fsharp.Reflection namespace, the functions there are not ideal since for example Spiral's tuples are not the same as F#'s. They get compiled to structs for one. Also with regards to records, there are multiple different types that use records and I have no way of differentiating them with just this.

What would be ideal for me if it would be possible to use some kind of System.Type token type that carries around metadata corresponding to the appropriate Spiral Ty. That would solve all my troubles. That, or a better alternative way of querying the .NET API.

I do not require a precompiled binary since I do not need to use all that System.Type offers like creating instances of types dynamically. Spiral's types cannot be represented in .NET anyway.

inl system = load_assembly .mscorlib
inl builder_type = system ."System.Text.StringBuilder"
inl b = builder_type ("Qwe", 128i32)
inl a x =
    b .Append x |> ignore
    b .AppendLine () |> ignore
a 123
a 123i16
a "qwe"
inl str = b.ToString()
inl console = ."System.Console" |> system
console .Write str |> ignore

inl dictionary_type = ."System.Collections.Generic.Dictionary`2" |> system
inl dict = dictionary_type(int64, int64)(128i32)
dict.Add(1,2)
dict.get_Item 1

Here is what the above example compiles into.

let (var_0: System.Text.StringBuilder) = System.Text.StringBuilder("Qwe", 128)
let (var_1: System.Text.StringBuilder) = var_0.Append(123L)
let (var_2: System.Text.StringBuilder) = var_0.AppendLine()
let (var_3: System.Text.StringBuilder) = var_0.Append(123s)
let (var_4: System.Text.StringBuilder) = var_0.AppendLine()
let (var_5: System.Text.StringBuilder) = var_0.Append("qwe")
let (var_6: System.Text.StringBuilder) = var_0.AppendLine()
let (var_7: string) = var_0.ToString()
System.Console.Write(var_7)
let (var_8: System.Collections.Generic.Dictionary<int64,int64>) = System.Collections.Generic.Dictionary<int64,int64>(128)
var_8.Add(1L, 2L)
var_8.get_Item(1L)

Maybe I should have asked this at the start, but what is F#'s type inference engine using to query the .NET methods? I have absolutely no idea, so it might be worth thinking in that direction.

That, or a better alternative way of querying the .NET API.

Yes I think that is what you are really after. It looks like customizing System.Type for your use-case (if possible at all) is a lot harder than what you actually need. All you need is to propagate your own info (type tokens) and then emit the proper stuff later.

One example is that you don't actually need the parameter types to query the available functions, all you need to do is to save your own map<param, "type token"> in addition to the generic type and query the generic version. This way there is no need to have System.Type objects for everything at compile time (in reality you only have them after compiling the F# code you are trying to generate).

So I think you need to get away from System.Type. You can use it for primitives and external libraries, however you should take a look at libraries like Mono.Cecil or https://github.com/jfrijters/Managed.Reflection. This by the way also solves the problem that you can currently only generate F# programs if you have references compatible for the runtime the Spiral-compiler is running on (which is probably not what you want).

The F# compiler brings it's own IL-Parser/Writer/Datastructures (I think it is https://github.com/Microsoft/visualfsharp/tree/master/src/absil).

One problem still remains: if you want to make "F#" libraries integrate nicely (ie. you want to detect records/tuples/discriminated unions of F# libraries) I fear that you need to look at the reflection APIs in FSharp.Core an adapt the logic to whatever reflection library you choose.

All the above needs to be taken with a bit of caution as it might not be 100% correct.

By the way Spiral looks nice ;)

I've looked at the two libraries you've recommended, but I do not know what to make of them. I do not think they are what I need. They barely have any documentation.

Last night I had a good idea that turned out to be unworkable. It occurred to me that I could use the WeakConditionalTable to attach metadata to type objects, but what tripped me up is that the System.Type instances are all globally unique for some reason which is surprising.

I guess I am going to do a dive into the F# compiler next.

In the worst case, all is not lost since you pointing out the FSharp.Reflection means that I can interop functions, tuples and heap allocated records easily now.

I am reluctant to building my own system for querying System.Type for the same reason I would be reluctant to do a directory browser. There is only a set amount of effort I want to dedicate to .NET integration and I do not want to go over budget.

I do not think they are what I need. They barely have any documentation.

They are "a better alternative way of querying the .NET API.".

I am reluctant to building my own system for querying System.Type for the same reason I would be reluctant to do a directory browser. There is only a set amount of effort I want to dedicate to .NET integration and I do not want to go over budget.

I can only warn you explicitly again: System.Type and in particular Assembly.Load and even Assembly.ReflectionOnlyLoadFrom have a lot of limitations you will encounter sooner or later. But yes they will give you the feeling of quick early success. We tried to go down that route in Paket and use Mono.Cecil now for basically everything.

So that I may better plan for the future, would you mind elaborating on the differences between Mono.Cecil and the System.Reflection? What are the limitations that you have encountered?

The examples are sparse, so it is not at all obvious to me what problems those two libraries you've mentioned address.

Well regarding the Assembly.Load, in this article it mentions that an Assembly cannot be unloaded once loaded using reflection and that Cecil takes care of that. So I guess there is no need to explain that.

Though the documentation on the Github repo is sparse, there are various tutorials online, so it does seem like I will find my way around. I've decided to take your advice. Rather than plan for the future, I'll figure out how to use Cecil now and level up my .NET metaprogramming along the way, though I honestly find this kind of programming nasty. This is why I've been so hesitant to commit to it.

Languages are made to get away from this sort of thing - in various ways this is a throwback to the days of assembly programming, only under a different guise. True metaprogramming is language creation.

I did not have any luck with Cecil. I asked the author how to get the methods with the substituted variables and got only silence in return which tells me that the library might not have been made for that sort of thing.

I do not have anything to choose from here. In terms of documentation, the little the AbsIL has in the F#'s compiler source is more than I've managed to find on Cecil.

Just now, I've noticed that AbsIL actually comes as a part of F#'s compiler services, so I've been trying to find some examples like tests on how it is used. I am thinking that it might be possible to use it to load an assembly and use it for queries. More generally and unrelated to the issue at hand, it would be interesting to generate F# AST directly and pass it to the compiler rather than text, but that is not possible right now as the library is mostly for the benefit of editors and IDEs.

Would it be possible to use AbsIL as an interface to .NET for my language? Specifically, I am making a request for some examples on how to load modules, query types and their methods and do substitution on generic parameters possibly with token types from my own language. Those token types can be just ints, it will be enough for them to be unique.

What I want here is really all straightforward and basic stuff, but the library is fairly huge and it is not at all readable to me, so I won't be able to do it without some help.

I've accepted @matthid 's advice not to use System.Reflection and am convinced that I need a better solution than it. It is really disappointing that .NET does not provide a built in solution to this given its size and complexity. In an article I've read, it was said that the .NET reflection functions were made to emulate the late binding of dynamic languages which is the reason for its unsuitability for other tasks.

If AbsIL cannot be reused, I will have to resolve myself to writing my own IL reader.

Probably, the best way to learn that stuff would be to make a .NET IL interpreter. What do you think?

Probably, the best way to learn that stuff would be to make a .NET IL interpreter.

I've just gone through the CIL Ecma spec and yeah, there is no way I am doing this. The one interpreter that I could find is insanely large at 12k lines of code. I was just grasping in desperation when I wrote this.

But at any rate, it does not matter as I've resolved how to deal with the interop issues, so please allow me to withdraw my selfish request for AbsIL examples.

Spiral has a pretty powerful type system, to the point of being capable of tracking the type's layout. I've only added that sort of feature just recently to the language and it only worked on functions so far, so it did not occur to me to extend it to tuples. There is also the matter that like in dependently typed languages, types can move around on the term level and can hold literals as well, so it did not occur to me that it might be possible to restrict that. It also did not occur to me that I could introduce a special layout for F# tuples and records which would allow me to use its reflection functions to construct the System.Type for them directly and pass it along. Not to mention when I was first grappling with the idea, I did not know they even existed until @matthid pointed them out in this very thread. Using System.Reflection would also allow me to use .NET methods dynamically at compile time when the arguments to them are solely literals which will be useful for loading schemas from files and using built in string functions and such. Setting the layout, plus heap or stack allocation is rather easy in Spiral - about as easy as it is to convert an int to a float in F#.

If I act on the above insights, that will get me 90% of the way there in terms of interop. I won't be able to pass along union types as they use type level strings for names, but the rest should be fine. If I ever want to write compilers in Spiral I should write my own tree based structures as they will be better optimized.

In hindsight the idea for this should have been obvious to me, but I do not think that inspiration comes out of the blue. It definitely has a search component to it; many of the simple features in the language took me a long time to work out as well, and this time was no exception.

@matthid, thank you for your help.

As an aside, the author did answer my question right now and it is not possible to substitute the variables with Cecil. If he can't then I certainly won't be able to .

@mrakgr I commented on the answer on SO.

I feel like you don't actually want Cecil to initiate some type handle for a generic type instantiation. All you want is the generic metadata (and this is all cecil can provide you with) the parameter replacement needs to be done later, but this will also enable you to emit code with generic parameters for types you define in Spiral (as long as they will be represented as types in the emitted F# code as well).

dotnet / fsharp

For a language made in F# that compiles to F#, how should more complex types such as tuples and records be integrated? #3723