How to use the "mapfields_as_maps" option

[tiagopog]

Hey there!

I'm coming from the Elixir's community in order to figure out a proper way to translate protobuf maps into Elixir maps with the Exprotobuf mix package.

Context

Exprotobuf uses the following functions to post process or scan/parse protos:

• :gpb_parse.post_process_one_file/2
• :gpb_scan.string/1
• :gpb_parse.parse/1

After parsing it defines the decode/encode functions for the records which in turn call the :gpb.{decode_msg/2,encode_msg/3} functions.

Problem

Currently, Exprotobuf doesn't support the option to use maps rather than list of tuples for protobuf maps, then I'm trying to figure out a simple way to pass it forward to the gpb.

I saw that there's this gpb_compile:file/2 function which accepts the mapfields_as_maps to be used via decoder/encoder generated to that record, but I couldn't find a clear documentation nor specs that shows how to achieve such feature.

Do you have any idea or example of how can I simply decode my protobuf maps into Erlang/Elixir maps?

... and thanks for supporting this awesome project :-)

[tomas-abrahamsson]

I see. With gpb, one could say that there are essentially two protobuf encoders/decoders: one data-driven in the gpb module, this is the old (original) one with no options at all, the second one is the generated code produced by the gpb_compile:file/2.

As a side note, the data to drive the encoder/decoder in gpb is the same as what is returned from gpb_compile:file/2 with the the option to_proto_defs. It essentially does the same the gpb_scan:string/1, gpb_parse:parse/1 and gpb_parse:post_process_one_file/2 that you described.

For development time reasons, I haven't spent any time implementing the same bells and whistles in gpb as in gpb_compile, but instead mostly kept gpb as a reference implementation so that I can use it for cross-comparing encode/decode results with the generated code. The generated code is (much) faster than gpb, it makes more sense to put efforts there.

A way to go, could be to change the Exprotobuf to call encode/decode functions in the generated code of in gpb. I've no idea how easy or difficult this would be, but it would bring access to the options.

I saw there's bitwalker/exprotobuf#61, don't know if it is best to continue discussion here or there?

[tiagopog]

Nice explanation, @tomas-abrahamsson, you have clarified some questions I had here.

I see there's a spec for the use case you mentioned above but there's also this gpb_compile_tests: compile_iolist/1 that does several things as it calls the gpb_compile:file/2 in a given point.

As I'm new to the Erlang world, could you please give me a simple example of how to use gpb_compile:file/2 + generated decoders/encoders to parse proto maps as Erlang maps for a sample proto file I'll show below.

There's no problem to use that spec as a base for the example, I just would like to see how it works without the overhead present in gpb_compile_tests:compile_iolist/1.

Sample proto:

// user.proto
message User {
  string name = 1;
  map<string, string> errors = 2;
}

I appreciate your help, cheers!

[tomas-abrahamsson]

Certainly. I suppose you would really want to tie this into your build process, but I'm afraid I don't know how enough about mix to help with that. Below is a transcript with erlang and shell commands. Let's say your user.proto is in the directory /tmp/example and gpb is built in $HOME/src/gpb

shell$ cd /tmp/example
shell$ ls -l
total 24
-rw-r--r-- 1 tab tab    84 mar 21 22:49 user.proto

shell$ erl -pa $HOME/src/gpb/ebin 
Erlang/OTP 19 [erts-8.2.1] [source] [smp:2:2] [async-threads:10] [kernel-poll:false]

Eshell V8.2.1  (abort with ^G)
1> gpb_compile:file("user.proto", [{i,"/tmp/example"},mapfields_as_maps]).
ok
2>   
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
^C
shell$ ls -l
total 24
-rw-r--r-- 1 tab tab 16053 mar 21 22:51 user.erl
-rw-r--r-- 1 tab tab   413 mar 21 22:51 user.hrl
-rw-r--r-- 1 tab tab    84 mar 21 22:49 user.proto
shell$ erlc +debug_info -I$HOME/src/gpb/include user.erl
shell$ erl                                                     
Erlang/OTP 19 [erts-8.2.1] [source] [smp:2:2] [async-threads:10] [kernel-poll:false]

Eshell V8.2.1  (abort with ^G)
1> rr("user.hrl").  
['User']
2> Bin = user:encode_msg(#'User'{name="abc", errors=#{"def"=>"x123", "ghi"=>"y456"}}).
<<10,3,97,98,99,18,11,10,3,100,101,102,18,4,120,49,50,51,
  18,11,10,3,103,104,105,18,4,121,52,...>>
3> user:decode_msg(Bin, 'User').
#'User'{name = "abc",errors = #{"def" => "x123","ghi" => "y456"}}

Some random notes:

• The -pa $HOME/src/gpb/ebin is to add a code path to the Erlang VM, so it can find gpb_compile
• The rr("user.hrl"). is just for the erlang shell to read the hrl file so it knows about the 'User' record. In an Erlang module, you would do -include("user.hrl"). instead.
• The erlc ... is just to compile the erl file, to get a .beam file. From inside Erlang (or Elixir) one would call compile:file/2 instead.
• The call to user:encode_msg/1 and user:decode_msg/2 does the serialization/deserialization work. See also the examples in the beginning of the README for more info.

Hope this helps you moving forward, and feel free to ask if there's anything more.

[tomas-abrahamsson]

I should maybe also say that the unit tests are not always a good example to look at. For example the compile_iolist is a bit convoluted, in part because I hadn't implemented gpb_compile:string at that time. Now most uses of compile_iolist could be turned into calls to gpb_compile:string plus perhaps some compiling and loading of the generate Erlang code.

[tiagopog]

Thanks for the awesome example, @tomas-abrahamsson!

I didn't test the whole thing yet – going to do that as soon as I get some free time – but I will close the issue since I don't have any other questions about this topic (for now :-).

Cheers!