msgpack-rpc is not compatible with other msgpack rpc implementations

[dgryski]

The fix to https://github.com/ugorji/go-msgpack/issues/10 that was carried over into the codec implementation was incorrect.

The 4th parameter to msgpack rpc is an array of all the arguments. However, Go requires that methods exposed through net/rpc have only a single argument. By having the "extra" array indirection on read and write, the "single argument" vs "array of arguments" problem is solved by passing an array consisting of a single argument, and both Go and msgpack are happy. This means that when using msgpack-rpc between Go and (say) Python, the more-restrictive calling conventions of Go (namely, only a single parameter) must be respected. The rpcCodec.ReadRequestBody routine can't be shared, it must be specialized. msgpackSpecRpcCodec writeCustomBody just needs to be patched.

You can test this against these trimmed-down client/servers in https://github.com/dgryski/trifles/tree/master/msgrpc

[ugorji]

I just looked into this, and am trying to figure out why I changed it previously in ugorji/go-msgpack#10 (ie https://github.com/ugorji/go-msgpack/commit/5d4be733d472935cbe3158b74990deebedba2a78)

@smw1218 do you have any thoughts? From looking again at the spec for like 30 minutes, it seems to me that @dgryski is right and I should not have made the changes previously. Can you point to what we may be missing? If not, I think the right fix is what was there in the first place (ie what @dgryski suggests).

[ugorji]

I have a fix in for this, and have tested it with your sample at https://github.com/dgryski/trifles/tree/master/msgrpc

A few things about your sample so it would work:

• Call RPC functions as Echo.Echo123 and Echo.EchoStruct. Without a . in the rpc request serviceMethod, we get an ill-formed argument error on the Go side. I'm not sure how to make it work compatibly with Python msgpack (which seems less flexible in allowing configuring a prefix for service methods).

The fix probably wouldn't go in for a few days because I was in the middle of adding very good support and performant support for BinaryMarshaler. (Supporting something like a BinaryMarshaler typically has a significant cost, as it involves extra conversions of reflect.Value to interface{} at top of each recursive function).

[dgryski]

Yeah, sorry about the state of the tests. I just commited my testing directory so I could have something to point you at -- both clients were set up to talk to the Python server. Too bad about the name-spacing issue though. I might see if there's a way to fix that from the Python side (just because I'm curious..)

[smw1218]

In order to be compatible with both the msgpack spec and the net/rpc lib, I think you needed to do some type checking. If the type passed in is a slice just pass it through. If not, wrap it as before.

That way, if a server requires multiple positional parameters, it's possible to pass an array in Go. But if the user passes in a non-array then it will wrap it in order to follow the spec.

[ugorji]

@smw1218 @dgryski

Perfect. Now I remember. The solution before wasn't robust enough. I think I finally see it fully.

We need to treat and handle the different calling conventions.

A Go Service requires that it only takes one argument. Services in other languages may take multiple arguments. However, msgpackrpc requires that the arguments get passed as an array.

The solution is to have a way for the calling client using our library to denote that a service taking more than one argument is being called explicitly.

The simple way, is to pass an argument of a specific type. I'm calling that type MsgpackSpecRpcMultiArgs.

package codec
type MsgpackSpecRpcMultiArgs []interface{}

Sample Code:

Calling a service in python:

    def Echo123(self, msg1, msg2, msg3):
        return ("1:%s 2:%s 3:%s" % (msg1, msg2, msg3))

Code using our library to call it:

    var reply string
    var args codec.MsgpackSpecRpcMultiArgs = []interface{}{"foo1", "foo2", "foo3"}
    err = client.Call("Echo123", args, &reply)
    fmt.Println("reply=", reply)

Within the codec, if the argument to client.Call is of type codec.MsgpackSpecRpcMultiArgs, we assume that you are calling a service method that takes multiple arguments and do not wrap the arguments in a slice of 1 again.

@dgryski

Thinking it through this way shows that services in other methods do not have the limitation of having a dot in their service name. Only Go RPC Server requires a dot.

I've tested this with changes to your code. Now I have

• cli-py.go (calling python service via go client)
• cli-go.go (calling go service via go client)
• cli-py.py (calling python service via python client)
• cli-go.py (calling go service via python client)
• srv.go (go service)
• srv.py (python service)

All iterations work.

I'd checkin the code when I do a push early next week. Please let me know if there is some urgency - I'm done but want to let the other refactoring I did stew for a bit.

[dgryski]

There's no urgency from my end.

[ugorji]

This is now fixed by commit c61e5837a8566751ab31434604dba9f23b126ce9