froto.exe code generator for proto3

[ctaggart]

I mentioned in #11 that I want to create an F# code generator that supports proto3. I want to make use of F# Records and Options. For basic serialization, either Google.Protbuf library can be used or I can see if Froto.Core functions I created earlier will be enough. Similar to using Roslyn to generate C# code, I want to use the F# AST to generate F# code. I've made quite good progress on FsAst, but it will probably need to be worked on to support this effort.

[ctaggart]

Thanks to @jhugard for revamping Froto.Core in #20, I'm going to generate code for it. I would love an example .proto and the code that should be generated for it, but I can probably figure that out.

[jhugard]

Should (hopefully) be easy to figure out, given the example in Froto.Core.Test/ExampleProtoClass.fs, but let me know if I should workup a .proto and equivalent .fs file.

Consider doing codegen for the Google descriptor.proto. It uses proto2 features that the official Google C# code gen cannot reproduce, and therefore can't be used to write code which consumes compiled protobuf definitions. Would make a compelling case for someone to give Froto a try...

[jhugard]

In ExampleProtoClass.fs, I made a mistake in the capitalization-conversion of the "id" field: this should be Pascal Cased as "Id" and not "ID" - unless you want to make all 2 character identifiers ALL CAPS, which might be OK.

[7sharp9]

Hi, I wondered what the current status is for froto.exe, does it currently generate and F# code? If so is it using the AST or roslyn etc?

[ctaggart]

@7sharp9 It doesn't exist yet. In my last three commercial projects, I've used https://grpc.io/docs/quickstart/csharp.html. It would be nice to have this Froto based generator. There has just never been a great way to generate F# code. ts2fable uses strings.

[7sharp9]

Its crazy there no real code generation available from F#. I think the closest tacky way of doing it is by using the untyped ast via quotations, and then serializing them back to code files.

[ctaggart]

Yes, that is the direction I was going with https://github.com/ctaggart/FsAst a couple of years ago

[7sharp9]

So the plugin for proto3 would have to take a CodeGeneratorRequest proto and produce a CodeGeneratorResponse as detailed here: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin.pb

[ctaggart]

Sure, that is if you want to make it a plugin for protoc where protoc reads in the .proto files. But Froto.Parser supports reading proto files, so you can skip making it a plugin.

Froto.Parser - Protocol Buffers Parser. Supports proto2, proto3, & gRPC syntax.

[7sharp9]

Yeah I mean it is easier to not implement it like that but Im thinking more in terms of a wider ecosystem, say your downstream clients are iOS and you want a swift definition generating. You would have to do Froto.exe blah.proto for F# then protoc --swift_out=. blah.proto for Swift

[ctaggart]

Correct, you could build it either way. There are several building blocks. Ideally, I'd love to see a dotnet 2.1 tool that you could installed named froto. froto blah.proto would then use Froto.Parser and generate code that used Froto.Serialization.

Another option would be to generate code that used Google.Protobuf, but F# code in a more F# friendly way.

[7sharp9]

Yeah Im planning on just generating F# code. Interesting to use the dotnet tool extension, i don't suppose it matters as long as the downstream systems have a way to generate the code for their language anyway. I'll see what I can come up with maybe this can be a PR later on...

[jhugard]

@7sharp9 My interest in Froto was to prove out some ideas for possible use with Unity that needed support for proto2 & proto3 both.

Time got short and I figured it might be faster to write the F# codegen in golang, leveraging the parser and validation in protoc, than it would be to write the rest of a compiler for Froto. I finished out just enough of an (unpublished) prototype to create record-based code for Froto.Core and Froto.Serialization. Didn't take all that long (but longer than expected).

Reducing compiled code size while retaining performance were my primary goals. Turned out the result was larger than that from Silent Orbit, probably due to my use of F# statically resolved type parameters and the resulting number of inlined functions. However, IIRC, the performance of Froto.Serialization was respective and competitive with other Protobuf libraries and supports Records to boot. On balance, probably worth considering for other use cases.

Upshot is writing a golang plugin for protoc, which generates Froto.Serialization code, is IMHO a very valid approach and probably a shorter path than the other alternatives.

[7sharp9]

@ctaggart I expanded on FsAst/froto/providedtypes and seem to be making some decent progress

[panesofglass]

@7sharp9 how's it going with the code generation? I was just thinking (again) of using this.

[7sharp9]

@panesofglass Its going ok albeit little slowly.

To be honest Im completely abusing the type provider code from froto and providedtypes.fs to produce F# source code rather than injected IL.

I currently have the capability to generate F# records from proto definitions with static and instance methods built from the quotations from froto. I have started modifying quite a few of them as the srtp architecture makes quotation splicing difficult.

[ctaggart]

The code generator is under active development at: https://github.com/jet/falanx

[7sharp9]

Currently working on Fleece serialization/deserialization support