Possibility of adding Go as a new backend

[maxnordlund]

Hi, I've stumbled into Mercury a couple of weeks ago and was very happy to see their is a strongly typed and moded logical programming language. I've was even more happy to see that it could compiled into high-level C, allowing it to be integrated into pretty much anything.

That got me thinking about using it with Go, and after thinking about it for a while I realized that it might be a good fit as a backend for Mercury. For those of you who haven't heard about Go its defining feature is the builtin concurrency primitives channels and goroutines, which basically are micro-threads. There are some other nice to have things, like garbage collection, exceptions, and returning multiple results.

This seams to fit Mercurys high-level C compiler infrastructure well. So my question is, do you think it would be a good backend language for Mercury? And if so how much work would it be to add it?

I'll of course help as much as I can, but I'm not a compiler person just yet. But for the Go parts I know then by heart now, since it's my current go-to language.

[PaulBone]

I'm pleased that you've found Mercury and that you're interested in it.

Appart from the development cost, adding a new language backend also adds maintainance costs. And to be honest, we struggle to maintain all our existing backends. Therefore, I think that it's important to have a strong justification for adding a new backend. Especially since Go is a very new langauge and although it's exciting right now, in five years time it may much less popular than the cool language of 2019.

Some of the possible reasons to add a new language backend are:

+ To enable interoperability with a particular environment (this is why
  we have Java and C# backends, to work with the JVM and CLR).

+ To provide better performance or functionality (compared with our
  existing backends), this is one reason there is some interest in a
  hypertheical LLVM backend, and why we experimented with the
  now-discontinued x86-64 backend.

+ To provide better cross-platform support.  This is the initial reason 
  why we wrote a C backend rather than an assembler backend.  It enables
  us to target different machines and OSs.

I assome that you're interested in interoperating with Go code/programs? In that case is it easier to use Go via one of the C backends than it is to write a new backend? The main barrier to this may be using two different garbage collectors in the same process. What if you write the Go and Mercury programs seperatly and communicate over network sockets?

Another option is to not use Go at all (or not use Mercury at all). It may simply be easier to write your whole program in one of the two languages and do without some of the features provided by one of the languages. In particular Go's authors didn't invent channels, coroutines and lightweight threads. Channels and threads were invented/formalised by Sir Charles Antony Richard Hoare (Tony Hoare) in 1978 (http://en.wikipedia.org/wiki/Communicating_Sequential_Processes). Mercury supports threads and channels so there's no need to use Go to use this feature. You could also use Erlang with Mercury's Erlang backend.

I hope this helps, thanks.

[maxnordlund]

I appreciate you taking time to answer so thoroughly. Yes, I realize that it would incur a substantial maintenance cost. But I'm willing to take on that responsibility, if provided with guidance and support from the rest of you.

Yes, channels/goroutines are of course not unique and the team behind Go mentions CSP as one of the mayor influences on the design of the language. I've also heard about this Haskell guy whipping out ~10 lines of code which implement Gos channels, but I haven't been able to find it.

I would like to write programs in both languages, using the strength of each when suitable. For example if I wrote a game I could use Mercury to handle AI, Go for network/state handling and maybe WebGL for drawing.

About calling C from Go and vice versa, there is something called CGO which does that. But it does not support cross compiling. Which is one of the nicest things with Go, I can compile to windows from my mac just using the standard distribution of Go. The other thing is that it is statically linked which means I can just send the resulting binary to the user and it will just run. As far as I can see this is not true for Java/C#/Erlang. They need a runtime as well. (Unsure about Erlang, dose it support static compilation?)

I could separate the two processes and talk over sockets/network, but in realtime applications such as games this has a to high latency cost. This is especially true for windows targets since they do not support BSD style sockets.

I also believe Go could allow performance increases due to it concurrent nature, at least for multi-core CPUs (are there any significant amount of single cores left?).

Last but hardly least there is the question about GC. I say that for Go it would be like for Java, let the targeted language handle GC. Depending on how you implement non-deterministic goals, it might be necessary to insert manual handling. I would use a channel as the "return", which you can read each possible answer from/write the next solution to. But this creates a memory leak because Go those not GC deadlocked goroutines, since cannot know if it will be needed again. Instead you need an extra close channel to signal downstream goals that no more answers are needed, see the blog post about piplines for more information. On the other hand this would give Mercury free concurrency which is always nice.

[wangp]

Presumably it would be another MLDS backend, like high-level C, Java, and C#. Adding the code generator shouldn't be too hard. You can look at compiler/mldsto*.m. The bigger problem, and maintenance burden, is in the standard library and RTTI implementations.

[PaulBone]

What I'm trying to suggest is that creating a new langage backend is probably more difficult (will take more of your time) than some other potential solutions, such as using Mercury's channels and not using Go at all. However I don't know what your needs are, so I cannot make this judgement.

I don't know about channels taking only ~10 lines to implement in Haskell, but Haskell does have channels and other concurrency libraries already. Like Mercury, haskell also has a good implementation of light weight threads.

Yes, C#, and Java both need runtimes. Erlang usually uses a runtime, I don't know if it's possible to compile it to a static binary.

I understand why using sockets isn't an option for things like games. That makes sense.

Does the other version of go (not CGO, then plan-9-ish one) have any kind of foreign function interface? How do developers using it call to arbitrary C libraries or call the OS's/libc's C functions. I would try to use that and Mercury's foriegn function interface together to have the two languages work together. Like I said in my other e-mail you may have to be carefull with regards to memory management. If you pass some Mercury memory to Go (or vice-versa) then the GC that created it may assume it's no-longer being used an unallocate it.

One developer is using this approach to integrate Lua and Mercury: https://github.com/C4Cypher/Apollo-lander

If you do want to write a new Mercury backend then Peter's reply should hopefully get you started. We will try to support you as best we can.

Thanks.

Paul Bone

[zsomogyi]

This proposal seems to have gone nowhere.