go2hx knowledge sharing and contribution

[PXshadow]

Background

In connection with the community post and some recent discussions on the matter of working together to improve Haxe's Go support in both directions.

Overview

This is a meta issue, for how can go2hx's tooling, knowledge, and compiler capabilities be used to improve haxego. As well as a place to discuss future joint ventures and developments effecting both projects for example Go and Haxe releases.

@elliott5 has also thought of building a Go target as well and has an abundance of expertise, he's also a go2hx contributor and can offer excellent ideas and advice.

Initial Ideas

• Compiling go/ast using go2hx for use in haxego.

• Add the ability for haxego to use go2hx's unit tests once compiled into Haxe code to test haxego's completeness. It might be useful because go2hx exports out most code that can be directly 1 to 1 remapped back to Haxe code.

• Use go2hx's github action system as a template to implement a CI runner for haxego, for example the regression checker system, the doc gen runner and more.

• Compile the entire Go std lib into Haxe externs for Haxe code to be able to use the Go std lib in Haxe in the same manner as the js package for Haxe's js target

• Allow haxego to call go2hx to generate automatic externs for go libraries for use in Haxe on the Go target.

• Setup haxego to use a similar system to go2hx's patch

• Copy over all of go2hx's std documentation over to haxego for use by Haxe devs.

• Add hooks into Haxego to use go2hxdoc to generate documentation for Go libraries used on the Go target of Haxe.

Feel free to add more ideas and let's create a discussion from this. I'm also interested @neimanpinchas what sounds appealing to you and in what priority.

[elliott5]

Hi @neimanpinchas, just to endorse what @PXshadow has said above, I'm hopeful that our Go/Haxe expertise might be of assistance to you on this project.

[elliott5]

To speed up the code generation, haxego could output the generated Go code as an AST (Abstract Syntax Tree) and have Go itself do all the formatting to get the detailed Go syntax correct.

There is at least one 3rd party open-source library to make this easier - https://github.com/tdakkota/astbuilders

I appreciate this would mean running Haxe and Go side-by-side initially, but once either haxego or go2hx becomes sufficiently mature, haxego can be transpiled into a single language. This dual-language approach is how go2hx is implemented (but in the other direction of course).

[neimanpinchas]

hi @PXshadow @elliott5

A lot of great ideas, I am doing hoime work on it

I hope to update the roadmap, github, in the next few days

Will think about priorities

I wonder if I could make some official scoring model on how to decide priorities

[elliott5]

A little example program to illustrate how to use Go AST to output properly formatted Go:

https://go.dev/play/p/nLK-6zsulba

[neimanpinchas]

Have you succeeded in converting https://github.com/tdakkota/astbuilders to Haxe?

[elliott5]

No @neimanpinchas we're still a little way off being able to do that I'm afraid.

So the argument against using Go AST is that it would add a perhaps unnecessary layer of extra complexity to the project. Especially given that badly formatted Go can be reformatted using a Go command line tool.

[elliott5]

FYI: go2hx uses JSON to transfer AST between the Go and Haxe halves of the project.

...it's a bit cleverer than that though, as I'm sure @PXshadow would explain if asked!

[neimanpinchas]

If we need to rely on the shell anyway, we could just run gofmt to achieve this, that way we could atleast skip this step if the user doesn't have go installed (so why is he using haxego).

Could be that with a few helper functions, AST generation will be cleaner than string concatation, in addition to the speed benefit @elliott5 mentioned

[PXshadow]

Yes indeed I would, but only if it'd be helpful 😅 , however I would hope that it would make more sense to compile go/ast and the dependencies listed below in the somewhat near future:

go list -f '{{ join .Imports "\n" }}' go/ast
bytes
fmt
go/scanner
go/token
io
os
reflect
sort
strconv
strings

Then writing another JSON AST transfer system over the local network, as @elliott5 has noted, unnecessary complexity could be a serious issue.

I also found that after using Haxe's own Printer in go2hx, I had gotten rid of an entire set of bugs, and I could focus then on AST transforms.

It was also really nice because Haxe code can use the macro keyword to create the AST by simply writing it with the macro keyword, the same fortune I don't believe is possible for haxego.

For example the easiest expr transform in go2hx looks like this:

private function typeStarExpr(expr:Ast.StarExpr, info:Info):ExprDef {
    var x = typeExpr(expr.x, info);
    final t = typeof(expr.x, info, false);
    if (!isPointer(t)) {
        final t = typeof(expr,info,false);
        final ct = toComplexType(t,info);
        return (macro ($x : $ct)).expr;
    }
    return (macro $x.value).expr; // pointer code
}

However the structure given by go/ast to create a CallExpr, and know that if you set the fields Fun and Args correctly you will get 100% valid Go code is very valuable:

type CallExpr struct {
    Fun      Expr      // function expression
    Lparen   token.Pos // position of "("
    Args     []Expr    // function arguments; or nil
    Ellipsis token.Pos // position of "..." (token.NoPos if there is no "...")
    Rparen   token.Pos // position of ")"
}

[PXshadow]

I agree with @neimanpinchas , I think string concatation is very error prone, and utility functions to create the AST is a huge improvement.

skip this step if the user doesn't have go installed (so why is he using haxego).

In the future haxego could use the Go compiler compiled into Haxe code by go2hx to remove the dependency need of Go, but that's really far in the future, I like the thought though!

[elliott5]

To summarise...

Both Go and Haxe have excellent well-tested AST encoding / manipulation / decoding tools as part of their standard libraries.

Using a language's native AST library is quicker to write, less error-prone, and faster at execution time than dealing with the text of a language's code directly.

Using the native AST library approach is a 3-phase process:

1. Encode source-language-code -> source-language-AST (using the source language AST library)
2. Transform source-language-AST -> target-language-AST
3. Decode target-language-AST -> target-language-code (using the target language AST library)