FFI with Maps or Containers

[CMCDragonkai]

The more complex data structures that require mapping between Haskell and Go shouldn't involve a map in the intermediate C. Whatever exists in C should be a very simple structure that carries pointers. Most likely an array of some sort.

I imagine a Haskell Map to a go function expecting a Go Map. Then the Haskell map needs to be serialised (as some sort of array of pointers), and those pointers must then be used to construct the Go map. However we have to be careful about mutability. If the go function mutates things not expected by Haskell, there could be segfaults. Further investigation here needed.

[CMCDragonkai]

Alternatively rather than directly sending a Haskell map to Go. Expose Go functions that mutate that Map back to Haskell, and Haskell only calls those functions rather than directly working on the Map. That reduces the amount of copying that exists, and keeps the memory local the Go environment.

I think this is probably the best way to do it.

[mokuki082]

I thought about doing exactly that, the problem is there are some restrictions to do with passing pointers in cgo. I cannot have a builder in Go that handles the creation and mutation of maps which at the end passes the pointer to haskell, and then get relayed back to Go at a later stage. Maps are one of the data structure that contain Go pointers within them, which could cause undefined behaviors when passed as a go pointer from C to Go.

[CMCDragonkai]

My understanding is that there are 3 ways to do this:

1. Maintain a Map in Haskell, work on the Map only in Haskell, and only when you really need to, pass that Map in bulk to Go for it to do work. (This is like doing as much as possible in Haskell).
2. Maintain the Map in Go, produce an abstraction/wrapper library that wraps the operations as much as possible in CGo. This depends on whether the Map is meant to be used as a get/set interface, or if the Map is an embedded data structure representing a more complex construct. For example in other languages I've often used the Map as an encapsulated state within a more higher order object, where the Object does things but uses the Map internally. Thus if the usecase of this Map is in this way, it may be possible to wrap the Map usage around more higher order functions.
3. A lens system that maintains isomorphism between the Haskell Map mutations and the Go Map mutations. Sometimes this is called "bidirectional transformation". See: http://www.seas.upenn.edu/~harmony/ Basically the idea is to perform state synchronisation. Think of Haskell Map and Go Map as databases, and you'll need to maintain replicated state between them.

To make this clearer, can you point out specific examples of Maps you want to FFI into, and what those Maps are used for? (Please add links to the source code with commit hashes, so Github comments embeds them).

[CMCDragonkai]

I split off the issue about bidirectional transformation to https://github.com/MatrixAI/Emergence/issues/33 I realised that this was an issue I thought about a long time ago that will be relevant later, hopefully we do not need to get that deep to solve this problem.

[mokuki082]

Closing this because we no longer bind into libcontainer, hence serialising map isn't the top concern.