Command-line argument to override the "optimize_for" option

[GoogleCodeExporter]

The project I'm working has C++ and Java code that need to share a few
.proto files. However, in C++, we only build the lite version of the
protobuf library, while the Java code needs to have the full version. Since
the "optimize_for" option can only be specified in the .proto files
themselves, that makes sharing of .proto files very difficult.

This is not specific to C++ or Java: in general, it may be useful to
codegen multiple source files with different optimization settings from the
same .proto file.

I suggest that protoc have an "--optimize_for" command-line argument that
overrides any such options in the .proto files. I'm attaching a patch that
solves this in a very hacky way; hopefully, someone will know the proper
way to fix this.

Original issue reported on code.google.com by d...@gmail.com on 12 May 2010 at 6:49

Attachments:

• optimize_for.patch

[GoogleCodeExporter]

Actually, the patch is wrong... It overrides optimize_for if it's set in .proto
files, but does not override the default value.

Original comment by d...@gmail.com on 12 May 2010 at 9:28

[GoogleCodeExporter]

Would it be much more difficult to support arbitrary option overrides on the 
command 
line?

Original comment by nick@chromium.org on 12 May 2010 at 9:35

[GoogleCodeExporter]

This is something I've always thought was needed. It makes much more sense to 
me to
define these kinds of options on the command line so you can easily switch them 
out
as part of a build process. This is especially useful when you are targeting 
multiple
languages, as mentioned in the original post.

Original comment by mike.wel...@gmail.com on 13 May 2010 at 7:50

[GoogleCodeExporter]

I agree, we should be able to override options on the command-line.  The only 
problem 
is that it's unclear how far this support needs to go.  Should you be able to 
override 
message-level and field-level options, or just file-level?  Should an override 
apply 
to an individual file, or should it apply to all the files it imports, too?  In 
the 
case of optimize_for, we'd probably want the override to apply to imports, but 
for 
something like java_package we probably don't.

Original comment by kenton@google.com on 17 May 2010 at 8:12

[GoogleCodeExporter]

We also need this feature. We're sharing .proto files between applications. One 
of the apps has a requirement to be optimized for code size and the other apps 
need to be optimized for speed. I don't care if arbitrary option overrides are 
supported but if they are it seems to make sense that they only be for 
file-level options.

Original comment by eshober@fusionio.com on 13 Jul 2010 at 9:28

[GoogleCodeExporter]

What about an even more generic implementation where you can optionally poke
arbitrary strings into the file from the commandline? So, for example, instead 
of
overriding options like optimize_for we could simply take a string from the 
command
line and prepend it at the top of the file something like this:

$ protoc --prepend-string "option optimize_for LITE_RUNTIME;"

Obviously, this only helps when there is not already an optimize_for option in 
the
file but this simple feature could handle lots of build issues like 
optimization.
The user would have to be responsible for using it judiciously.

Comments?

Original comment by eshober@fusionio.com on 16 Jul 2010 at 7:52

[GoogleCodeExporter]

If you just want to append a string to the file, you can easily do that like:

  cp foo.proto foo2.proto
  echo 'option optimize_for LITE_RUNTIME;' >> foo2.proto
  protoc foo.proto

Any reasonable build system should allow you to do this without changing protoc.

Original comment by kenton@google.com on 20 Jul 2010 at 9:44

[GoogleCodeExporter]

Now suppose foo.proto is imported by a few other .proto files... You'll have to 
modify each one of them, too.

Every other compiler - gcc, javac, etc. - lets the user specify the 
optimization level on the command-line. Why not protoc?

Original comment by d...@gmail.com on 20 Jul 2010 at 9:52

[GoogleCodeExporter]

I agree that protoc should provide something.  I was merely pointing out that 
eshober's suggestion can trivially be implemented without protoc's help.

Note that the optimization option for gcc and javac is very different from lite 
vs. non-lite.  When you compile a C++ file with -O2, the compiler doesn't need 
to know what optimization level was used for all the other files you intend to 
link against -- it doesn't make a difference.  But protoc needs to know not 
just whether the file it is compiling is using LITE_RUNTIME, but also whether 
all the files imported by that file are using it.

This makes the problem significantly more complicated, because if you specified 
the optimization options for each of those imports on the command line when you 
compiled them, then you have to somehow express the same information on the 
command line for the dependency.  This could get really awkward:

  protoc foo.proto --option=foo.proto=optimize_for:LITE_RUNTIME
    --option=some_other_package/bar.proto=optimize_for:CODE_SIZE
    --option=yet_another/baz.proto=optimize_for:SPEED

To make matters more complicated, imagine this situation:  You are linking 
against a library which includes foo.proto.  The library compiles in this proto 
in lite mode.  You, however, want to import the proto into a file that is 
compiled in regular mode (and you actually need reflection, so you can't just 
switch to lite).  This means you need to compile a copy of foo.proto into your 
binary, compiled with different options.  Because you are linking against a 
library that already has the lite version compiled in, you need to make sure 
that your copy doesn't have conflicting symbol names, and so you probably want 
to alter the package name too.  But, you don't want to maintain your own 
parallel foo.proto with these changes -- you really want to take the one from 
the other project and apply some command-line options to alter it.

What sort of command-line syntax would support this?

My concern is that a lot of people think this problem is trivial, but don't 
realize that the trivial solutions they are thinking of don't actually work, or 
only work for a smallish subset of cases.  I don't want to introduce something 
that we'll just have to immediately deprecate and replace with something more 
general.

Original comment by kenton@google.com on 20 Jul 2010 at 10:19

[GoogleCodeExporter]

Admittedly, my suggestion above does not handle imports and that makes it 
mostly unusable whether done in or outside of protoc. The assumption on my part 
was that the line inserted would affect all imported files like a C header 
include directive which simply inserts the text from the included file at that 
spot.

The only thing that makes sense to me and that doesn't get too complicated is 
that we add a command-line optimization option override that applies to the 
specified file and all imported files.

   protoc foo.proto --optimize_for=LITE_RUNTIME

I would need to use this to create a lite and a non-lite library so that my 
applications can choose which to link in without duplicating .proto files.

Original comment by eshober@fusionio.com on 21 Jul 2010 at 1:19

[GoogleCodeExporter]

Hi all,

Attached is a little protoc plugin written in Python which converts all the 
inputs to LITE_RUNTIME -- including renaming so that they do not conflict -- 
and then passes them on to some other plugin.  See readme.txt in the archive 
for details.  Basically with this you can do:

  protoc --litify_out=cpp:. foo.proto

And this will produce foo_lite.pb.h and foo_lite.pb.cc, which contain foo's 
types compiled with optimize_for=LITE_RUNTIME.  You can do Java too:

  protoc --litify_out=java:. foo.proto

You can easily modify this plugin's code to produce any arbitrary 
transformation you want.  It's just a Python script that processes the 
FileDescriptorProtos representing the inputs.

I like this approach because it is completely general.  There is nothing that 
you can't express in a Turing-complete language.  In contrast, *none* of the 
other proposals made so far can express even what this simple plugin does (in 
particular no one has proposed a solution for the renaming part).

I also like that this requires no modification whatsoever to protoc.  Not just 
because I want to avoid work, but because keeping the code code as simple as 
possible ensures that it retains agile.

What do you think?

Original comment by kenton@google.com on 28 Jul 2010 at 6:11

Attachments:

• litify.tar.gz

[GoogleCodeExporter]

s/code code/core code/

s/retains/remains/

*sigh*

Original comment by kenton@google.com on 28 Jul 2010 at 6:13

[GoogleCodeExporter]

Original comment by kenton@google.com on 17 Apr 2011 at 7:47

[GoogleCodeExporter]

I was able to use Keaton's plugin as a starting point and am using it now.  I 
didn't know that the protobuf compiler was that extensible.  Way cool.

FYI, it is less lines of code to write the entire plugin in C++ and directly 
call the C++ generator from there, though it is a bit trickier to get it right.

Original comment by austin.l...@gmail.com on 5 Nov 2014 at 11:38