GWT-compatible Protocol Buffers compiler

[dankurka]

Originally reported on Google Code with ID 2649

Found in GWT Release: all

Detailed description:

Google has recently released Protocol Buffers (a.k.a. protobufs) as an
open-source project. This project is an efficient cross-language message
exchange framework between applications (or multiple tiers of one
application). Message exchange is a big part of GWT too. It takes place
during RPC calls from the client application to the server. As of now the
Java code generated by the protobufs compiler is not compatible with GWT
compiler, because it uses Java reflection. Consequently it is not possible
to serialize/deserialize messages on the client-side of a GWT application.
Inability to do so requires one to manually create a translation layer to
translate from protobufs messages to GWT serializable objects and back. It
also creates a certain degree of incoherence between GWT and protobufs
(both are Google-lead projects).

Java reflection is used in protobufs to achieve two goals:

- Smaller code size
- Message introspection

There may be other ways for GWT to achieve smaller code size. GWT compiler
already has a very effective JavaScript optimizer. Message introspection is
just one of many use-cases for protobufs and can be either omitted or
implemented without reflection if possible.

I propose that GWT implements a compiler for *.proto files that generates
GWT-compatible Java source code. For better code-reuse it might be possible
to implement it as an option for the default protobufs compiler, but this
will require a higher level of collaboration between the teams.

Links to the relevant GWT/Protocol Buffers Developer Forum posts:

http://groups.google.com/group/protobuf/browse_thread/thread/32fb6405b8f6a337/0ddaf4ec9d972ba2#0ddaf4ec9d972ba2

Reported by Yegor.Jbanov on 2008-07-09 17:09:42

• Blocked on: #5499

[dankurka]

It would be really nice to be able to use protobuffs to communicate between gwt and
app engine. Hope you add support soon.

Reported by mauzepeda on 2008-07-28 06:40:50

[dankurka]

I agree, this would be awesome.  I wish we had the bandwidth to pursuit this right 
now as a first priority; it would be great if someone picked up the ball on this and

contributed a patch.

Here is a possible strawman design:

1) An external tool to read a .proto file and generate a Java interface; this is 
necessary to do up front because otherwise what a client write code against?  
Actually, a tool like this might already be part of the protocol buffer project, or

could potentially become part of it.  Ideally, the generated interface would contain

all the information in the .proto file, possibly somewhat encoded in metadata.

2) A GWT generator to produce a client side serializtion/deserialization factory.

Here's a strawman design:

// Generated from .proto file:
interface MyProtoBuf /* extends ProtoBuf marker interface? */ {
  // getters, setters, etc
}

interface MyProtoBufSerializer extends ProtoBufSerializer {
  String write(MyProtoBuf);
  MyProtoBuf readMyProtoBuf(String);
}

3) Some kind of server support.  One idea is to convert between String and bytes, and

use the built in protobuf classes on the server.

----

Alternatively, someone could just write an external tool to produce a protobuf 
implementation that a) doesn't use reflection and b) natively uses String as the 
serialized form.

Reported by scottb+legacy@google.com on 2008-07-29 19:22:29

• Status changed: Accepted
• Labels added: Type-Enhancement, Priority-Low, Category-RPC
• Labels removed: Type-Defect, Priority-Medium

[dankurka]

Here's a tutorial on using the existing Java tools:

http://code.google.com/apis/protocolbuffers/docs/javatutorial.html

It works like Scott describes: you compile .proto files to .java files and then use
the java files to access, serialize, and 
deserialize the protobuffers.

The open question is whether or not the generated Java files work in GWT.  Has anyone
tried?

Reported by spoon+personal@google.com on 2008-07-30 16:53:37

[dankurka]

I have no doubt that it will not work. Some core framework classes have references to
java.io, java.nio and reflection packages and implement lots of convenience methods
to support stream-based communication. GWT compiler will choke on this stuff.

Naively, I went through the source code and tried to remove all these references, but
it turned out to be a hard task (at least not something that can be done in a matter
of minutes). I am almost positive I broke functionality in the process. Needless to
say, the files generated from the sample *.proto files were glowing red in Eclipse
after all this. Even the message as simple as the Person (from the project's main
page) with option optimize_for=SPEED did not allow an easy fix.

I know my mistake. I should have started with removing ONLY java.io and java.nio and
should have kept the reflection for now...

I'll keep in touch.

Thanks,

Yegor

Reported by Yegor.Jbanov on 2008-07-30 21:10:40

[dankurka]

I'm actually kind of surprised they use runtime reflection.. what in the world would

you need it for?

After looking at this a bit, I'm wondering if the best solution wouldn't be an 
extension to the protobuf compiler for GWT.

Reported by scottb+legacy@google.com on 2008-07-30 23:22:58

[dankurka]

http://code.google.com/p/gwt-protobuf/

Reported by scottb+legacy@google.com on 2008-08-15 23:42:53

[dankurka]

Reported by bruce+personal@google.com on 2008-10-21 21:54:59

• Labels removed: priority-low

[dankurka]

Reassigning to Bob.

Reported by scottb+legacy@google.com on 2009-03-10 20:03:48

[dankurka]

wondering what the status is on this issue?  is it a low priority for team GWT?

also, is the link on comment 6 restricted or just broken?

Reported by davidramsayreinhart on 2009-07-10 19:00:26

[dankurka]

Low priority, basically.  As for the link, for some reason the owner decided to abandon

the project and delete it.

Reported by scottb+legacy@google.com on 2009-07-10 22:07:05

[dankurka]

Interesting that the link returns HTTP 403 Forbidden instead of HTTP 404 Not Found,

like http://code.google.com/p/nonexistentproject42536467

Just thinking aloud...

If I remember correctly, someone mentioned at Google I/O that the Wave team created
a 
custom Protobuf compiler to do RPC. Maybe it could be extracted into a separate project

once the thing is open-sourced.

Reported by Yegor.Jbanov on 2009-07-10 22:18:54

[dankurka]

Is this even practical because of the limitations of JavaScript using binary formats?
I 
would like this too, but it seems converting protocol buffers to json on the server

would be the only practical way to approach this.

GWT compiles to JavaScript and inherits some its limitations. According to http://www.w3.org/TR/XMLHttpRequest/
binary streams are yet to come to JavaScript.

Reported by datagrove on 2009-08-23 22:57:47

[dankurka]

I'd be happy enough if it just supported Serializing to/from normal protocol buffers

via ASCII, as long as it supports GWT RPC in its place.  Perhaps compile 'message 
Foo' into 'class Foo' and 'class FooGWT' along with a translator?

I just want to add new fields to my types with minimal total work and be able to use

them in both client and server.  Right now I'm contemplating writing my own proto-
>java compiler that just generates simple classes ('FooGWT') full of public data 
members, just so I can continue to define my fields in one set of .protos instead of

some .protos and some Java classes.  I imagine other people have similar desires.

Reported by mschmit on 2009-08-23 23:26:31

[dankurka]

@datagrove: Implementing the binary encoding of protobuf is not necessary. As you 
yourself pointed out, JSON might be a good solution. After all protobuf is many things:

- A message definition language
- A service definition language
- A Java API
- A binary encoding
- Python API, C++ API and other things that I forgot to mention

Of these, if a GWT-compatible compiler can keep the first three (or even the first
and 
the third), substituting the rest with whatever makes sense in a web-browser context,
it 
would already be a very useful tool.

@mschmit: I am with you on this one.

Reported by Yegor.Jbanov on 2009-08-24 04:40:20

[dankurka]

We've open-sourced our Thrift-to-GWT compiler as part of a GWT RPC enhancement
project recently (http://code.google.com/p/gwt-rpc-plus/).  The gwt-rpc-plus project
uses raw JSON as a transport, such that RPC responses are directly evaluated to
JavaScriptObject subclasses (example:
http://code.google.com/p/gwt-rpc-plus/source/browse/trunk/gwt-rpc-plus/example/gen-java-gwt/com/dotspots/rpcplus/example/torturetest/client/SimpleObjectWithFieldIds.java).

It would be reasonably simple to layer a protobuf compiler on top of the base RPC
support we've implemented.  Matching the API of protobuf's output wouldn't be
terribly difficult either.

If anyone is interested in going down this path, let me know.

Reported by mmastrac on 2009-08-24 05:21:59

[dankurka]

I have implemented a, crude, but working 'complier' to turn protobufs into GWT ready
java.  (see example attached).  Though I would caution, parsing 'large' (I tested
3500 protobufs containing a String and ~7-10 floating point numbers) can start to
choke some browsers.  Chrome and Safari worked great, but FF took much longer (about
18 seconds) and IE, well, IE did it's thing... over 1 minute.

Anyway my point is parsing the protobufs server side and sending them in JSON turned
out to be MUCH faster.  With all browsers having a reasonable JSON parsing time
(under a second).

I then just modified my 'compiler' to spit out GWT overlays for the JSON instead of
the proto classes.

Reported by jrmerz on 2009-08-25 17:42:23

---

• Attachment: CimisProtos.java

[dankurka]

You might want to check http://www.bibale.com/bibale/gwt-gwtready.patch which is a
patch to the latest thrift compiler to be able to use the generated Java objects in
GWT.

Reported by Mathias.Herberts on 2009-10-05 21:08:20

[dankurka]

I was watching the Google IO 2009 Video about how they made Google Wave in GWT among

with the problems they came accross and how they solved them. Very insightful 
information, and they mention they used protocol buffers, I wonder how they did it,

or if they could open source that part of the project.

Reported by mauzepeda on 2009-10-05 21:33:52

[dankurka]

Maybe using http://code.google.com/p/protobuf-java-format/ or a similar approach; as

they also talk about their need for JSOs to implement interfaces (protoc generating

interface + JSO for the client side + "normal" Java protobuf message for the server-
side).

Reported by t.broyer on 2009-10-05 21:37:43

[dankurka]

Reported by scottb@google.com on 2010-02-03 17:03:26

[dankurka]

See also http://code.google.com/p/protostuff and (shameless plug) 
http://code.google.com/p/protobuf-gwt

Reported by t.broyer on 2010-05-05 23:00:05

[dankurka]

what about 'not so long ago' released google wave? google wave is written in gwt and
uses protobuf.
and also i noticed that gwt-dev.jar contains protobuf-lite

Reported by nordlig.ulv on 2010-09-14 12:25:49

[dankurka]

AFAICT, http://code.google.com/p/wave-protocol uses protostuff (linked above). gwt-dev
uses protobuf to talk to "remote UIs", such as the DevMode view in the Google Plugin
for Eclipse.

Reported by t.broyer on 2010-09-14 16:34:25

[dankurka]

Reported by bobv%google.com@gtempaccount.com on 2010-10-28 04:21:42

• Blocked on: #508

[dankurka]

Reported by rjrjr@google.com on 2011-01-13 03:09:44

• Labels added: Type-Feature
• Labels removed: Type-Enhancement

[dankurka]

I have been coming back to this thread for the past six months in a hope that someone
would come up with a protobuf solution that would allow me to reuse protobuf-compiler-generated
Java class definitions on the client and the server side. That is, a solution that
would go beyond simple serialization of protobuf messages to XML/JSON. 

In the end, faced with hand-coding bunch of protobuf definitions on the client side,
I modified the protobuf compiler by adding a new gwt_out mode that outputs Java protobuf
class definitions very similar to what you would get with java_out but that do not
rely on reflection and can therefore be fed to the GWT compiler. In addition, I replaced
native binary protobuf encoding methods with serialization methods to and from JSON.
Also, having added GWT support largely for my own selfish reasons, I left unimplemented
some less-popular protobuf features such as groups, services, and, more importantly,
extensions.

The new GWT-friendly protobuf class definitions can be used both on the client and
on the server, but because they do not support native binary encoding, they are different
from their native Java equivalents. You can, however, use GWT-friendly and native protobuf
definitions side-by-side and convert between them on the server-side by using reflection.
This way, you can use the native version of a protobuf with its efficient encoding
for things like storage and use the GWT version for moving the data between the client
and the server as well as reusing protobuf class definitions on the client side. I
added a new gwt_package option that can be used alongside java_package to keep native
and GWT-friendly protobuf class definitions in different packages.

This is a new implementation, and as such it has not been extensively-tested, and comes
with little additional diagnostic functionality, especially around unsupported features.
Use it at your own risk. That said, it works great with my quite substantial code base.
If you think this is something you could benefit from, I posted the code at http://code.google.com/p/gwt-friendly-protobuf/
with more detailed information and build instructions on my blog at:  http://www.vitaliykulikov.com/2011/02/gwt-friendly-protocol-buffers.html

Hope this saves someone a lot of time.

Reported by vkulikov on 2011-02-28 01:33:00

[dankurka]

Reported by rjrjr@google.com on 2011-02-28 20:37:30

• Labels added: Milestone-2_4

[dankurka]

http://code.google.com/p/google-web-toolkit/wiki/ProtocolBuffers

Reported by bobv%google.com@gtempaccount.com on 2011-03-21 17:27:36

[dankurka]

Reported by bobv%google.com@gtempaccount.com on 2011-06-23 21:11:12

• Labels removed: Milestone-2_4

[dankurka]

Reported by rjrjr@google.com on 2011-10-05 00:07:19

[dankurka]

This has been put on hold. Which is not to say it will always be on hold, but please
don't make plans on it.

Reported by rjrjr@google.com on 2011-10-05 00:44:17

• Status changed: PatchesWelcome

[dankurka]

Has there been any progress on this? It has been more than a year since the last update.
Our research team is very interested in having this.

Reported by vidal@umich.edu on 2012-11-01 04:26:31

[dankurka]

This looks very useful, how soon can we expect it?

Reported by hamham91 on 2012-11-18 19:56:55

[MartinTrummer]

now that proto3 supports json mapping, is there a chance to see some new serialization in gwt 3.x?

[tbroyer]

Probably not “in GWT 3.x”, but there's no reason it couldn't be done as a separate library (which doesn't preclude compatibility with GWT 2.x).