chrislusf / glow

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
3.2k stars 248 forks source link

[feature] use github.com/DeDiS/protobuf for ser/de operations #8

Open chrislusf opened 8 years ago

chrislusf commented 8 years ago

The affected code is only within this file:

https://github.com/chrislusf/glow/blob/master/io/channels.go#L77

and

https://github.com/chrislusf/glow/blob/master/io/channels.go#L54

I tried to use github.com/DeDiS/protobuf, but it can not handle reflect.Value.

joeblew99 commented 8 years ago

i was using protocol buffers (PB) and recently started to use FlatBuffers (FB) for perf reasons. FB is the successor to PB that has been in the alpha oven for a while now, but is now ready (IMHO)

http://google.github.io/flatbuffers/

if seems a good match this Glow, and i am curious on your thoughts. There is no reflection, but instead the go files are code generated at compile time. There is no heap if used correctly at runtime.

I also liked the ability to serialise between FB and JSON. This allows the data types to be reused in the any web layer for a application.

chrislusf commented 8 years ago

Protocol Buffer or Flat Buffer need definition and code generation. They could be used to customize ser/de.

The default ser/de should not need any additional steps.

joeblew99 commented 8 years ago

ok. I need to look into the code to see how your doing it. I guess its reflection based ?

alpe commented 8 years ago

Fyi: The package was renamed to netchan: https://github.com/chrislusf/glow/blob/master/netchan/channels.go#L52 https://github.com/chrislusf/glow/blob/master/netchan/channels.go#L77

zenixls2 commented 7 years ago

if you really want to use DeDiS/protobuf, maybe write a func EncodeValue(value reflect.Value) to wrap from L51~L57 https://github.com/dedis/protobuf/blob/master/encode.go#L51 and write another func (dec *Decoder) DecodeValue(v reflect.Value) to wrap L64 https://github.com/dedis/protobuf/blob/master/decode.go#L64