2 new compilers: proto_extender, java_bean_model

[GoogleCodeExporter]

The question is more about how should i do that? 
Should i send you a compiler patches, or i should create separate github 
compiler project?

We have developed 2 compiler extensions for protostuff-compiler to support our 
needs at Neodinamika.

1) proto_extender compiler is the compiler, that allows one to help with 
inheritance problem in proto files. It can compile for you new proto files in 
which some Messages are directly extended by fields from other Messages.

So if you have:

message A {
 @Annotation(...)
 field int32 id = 1;
}

@Extend(by = A)
message B {
 field string name = 100;
}

You will get:

message B {
 @Annotation(...)
 field int32 id = 1;

 field string name = 100;
}

2) java_bean_model is the extension of java_bean compiler. The most important 
thing it have, is producing of inheritable schemas. More over that, even writes 
can be overriden. Repeated fields can use base classes for it's collections to 
support some kind of inheritance in that fields. Also renaming supported, and 
custom functional checks on writes/reads. It also generates fields using 
accessors methods is/get/set to provide you an access to fields to your 
existing models. Also fields can be annotated to support setting/getting from 
byte[],short,byte,char fields. This allows you not to change you existing 
schemas. 

with this functionality we were able to made following:

message Item { ... }

@Extend(by=Item)
message Weapon { ... }

@Extend(by=Item)
message Pet { ... }

@Extend(by=Item)
message Drug { ... }

message Hero {
 @Collection(...)
 optional repeated Weapon weapons = 1;
 @Collection(...)
 optional repeated Pet pets = 2;
 @Collection(...)
 optional repeated Drug drugs = 3;
}

but the model contains:

public class Hero {
 public Collection<Item> getItems();
 public void setItems(Collection<Item> items);
}

Original issue reported on code.google.com by John.Ko...@gmail.com on 25 Mar 2012 at 10:48

[GoogleCodeExporter]

Cool.  This is a great way to use annotations.
You can attach the patch here (assuming your changes are based from trunk and 
not 1.0.4).

Thanks!

Original comment by david.yu...@gmail.com on 25 Mar 2012 at 11:44

• Changed state: Accepted

[GoogleCodeExporter]

Yes, changes based on trunk 1.0.5-SNAPSHOT.
Will attach them a little bit later on a week.

Original comment by John.Ko...@gmail.com on 25 Mar 2012 at 1:08

[GoogleCodeExporter]

This is a first part. proto_extender compiler. i guess it should be added to 
compilers list in main protostuff-compiler class.

As you will see it supports both @Extend(by = ), and option extends = ...; 
syntax.
When you will try to build something with maven plugin using this compiler, it 
will try to keep same directories structure.

The second part (java_bean_model) will be a bit later on a week.

Original comment by John.Ko...@gmail.com on 26 Mar 2012 at 11:05

Attachments:

• ProtoToProtoCompiler.java
• proto_to_proto.stg
• extender_sample.proto
• extender_result.proto
• extender.properties

[GoogleCodeExporter]

Thanks for the first part.  I'm interested to see the second part and its 
output.
1.0.5 will be released by the weekend.  It would be good if this gets included.

Original comment by david.yu...@gmail.com on 28 Mar 2012 at 3:15

[GoogleCodeExporter]

Ok, will try to commit it tomorrow...

Original comment by John.Ko...@gmail.com on 28 Mar 2012 at 3:30

[GoogleCodeExporter]

So, here it is what we got right now.

The main goal of this compiler to provide you a flexible schema generating 
making you possible to inherit it and to change it's behavior.  

It can also right now produce separate model classes currently much like 
java_bean default compiler does it. This option do not implement a lots of 
features it should:
- immutable messages models (as option)
- copy()
- builder
- toString()
- native types for fields (support for existing annotation in schema @Byte, 
@Bytes, @Short, etc)
This functionality is not implemented yet.

Resulting schemas are similar to what can be got from java_bean separate 
schema, but more flexible. 

The second goal of this schema compiler was to provide you a total separation 
of your existing models and generated schemas that can be also reusable. That 
is why there are options like _package (model prefix) and _suffix (model 
suffix). And this is a reason why all schema classes generated with Schema 
postfix.

The third goal was to handle inheritance problem in our existing models. There 
are 2 kinds of inheritance problems:
1) instantiating of model it self (that is what returned by newMessage() in 
schema declaration, or what is returned by mergeFrom message field) 
2) instantiating of models that lies in on base class collection.

Any kind of this problems can be more or less handled by this schema generator 
with a help of schema inheritance support.

Original comment by John.Ko...@gmail.com on 29 Mar 2012 at 1:41

Attachments:

• ProtoToJavaBeanModelCompiler.java
• java_bean_model_base.stg
• java_bean_model_model.stg
• java_bean_model_schema.stg

[GoogleCodeExporter]

I have no enough time right now to provide ready sample how any one can handle 
inheritance with this. But this is the sample of features it supports.

Original comment by John.Ko...@gmail.com on 29 Mar 2012 at 1:43

Attachments:

• sample.proto
• modules.properties
• ClubModelSchema.java
• ClubFounderModelSchema.java
• ClubModel.java

[GoogleCodeExporter]

The main thought to get it: you can inherit generated schemas to override 
creation of fields with needed types.

message Item { ... }

@Extend(by=Item)
message Weapon { ... }

@Extend(by=Item)
message Pet { ... }

@Extend(by=Item)
message Drug { ... }

message Hero {
 @Collection(name = "items", type="Item", checkFunction="isWeapon")
 optional repeated Weapon weapons = 1;
 @Collection(name = "items", type="Item", checkFunction="isPet")
 optional repeated Pet pets = 2;
 @Collection(name = "items", type="Item", checkFunction="isDrug")
 optional repeated Drug drugs = 3;
}

but the model contains:

public class Hero {
 public Collection<Item> getItems();
 public void setItems(Collection<Item> items);
}

public class SuperHero extends Hero {
}

you can override schema to provide you with inheritance of deserialized type.
you can override schema to provide you with inheritance of models in 
collections.

Original comment by John.Ko...@gmail.com on 29 Mar 2012 at 1:53

[GoogleCodeExporter]

more over of that. you can use following hacks to provide you with even more 
inheritance:
1) for base type A that have 2 subtypes B and C you can use and transparently 
provide to your schemas following: message ASubType { byte sub_type_class = 1, 
optional B = 2, optional C = 3 }. you can override schemas to get to your 
models class SomeModel { private A baseClassPtr; ... }
2) you can even try to include "type of subclass" field as the first field of 
the message and to try to hack the protocol (with peek few first bytes) to get 
to know the type of needed instance to be created during deserialization 
process by it's schema.

The final schema for mentioned sample could be looked like this:
public class HeroSchemaFinal extends HeroSchema {
    static final Schema<Hero> SCHEMA = new HeroSchemaExt();
    public static Schema<Hero> getSchema() { return SCHEMA; }

    @Override
    public boolean isWeapon(Item item) {
        return item.getItemType() == ItemPrototype.TYPE_WEAPON;
    }

    @Override
    public boolean isPet(Item item) {
        return item.getItemType() == ItemPrototype.TYPE_PET;
    }

    @Override
    public boolean isDrug(Item item) {
        return item.getItemType() == ItemPrototype.TYPE_DRUG;
    }

}

Original comment by John.Ko...@gmail.com on 29 Mar 2012 at 2:04

[GoogleCodeExporter]

There seems to be a bug.

            case FIELD_RP_ABYTES_NO_CHECK:
                if (message.getRpABytesNoCheck() != null) {
                    for (byte[] rpABytesNoCheckEntry : message.getRpABytesNoCheck()) {
                        if (rpABytesNoCheckEntry != null)
                            output.writeByteArray(FIELD_RP_ABYTES_NO_CHECK, rpABytesNoCheckEntry, false); // isn't this supposed to be true?
                    }
                }

// here's the declared fields from the sample proto.
    repeated bytes rpABytesNoCheck = 9;
    repeated bytes rpBytes = 10;
Note that rpABytesNoCheck uses byte array.  And rpBytes uses a ByteString.

In my own pojos, I use byte array directly as well so that is a good 
optimization.

Is this patch direct from your build of protostuff? or from scratch (you 
created one for this issue)?

Original comment by david.yu...@gmail.com on 29 Mar 2012 at 2:36

[GoogleCodeExporter]

Yes it is build of protostuff from our project.
I created it to handle inheritance issues and byte[] problem in our models.

Yes i think you are right. It seems there should be true instead of false in 
repeated fields. I guess i missed it. The real reason why i missed it is cause 
this boolean repeated argument is not used in ProtobufOutput serialization 
implementation. 

You can fix it if you want of course. ))

Original comment by John.Ko...@gmail.com on 29 Mar 2012 at 2:47

[GoogleCodeExporter]

What about the ByteString and byte array.  Do you use both?
If I understand correctly, the @NoCheck annotation uses byte array instead of 
ByteString yes?

            case FIELD_RP_ABYTES_NO_CHECK:
                if (message.getRpABytesNoCheck() != null) {
                    for (byte[] rpABytesNoCheckEntry : message.getRpABytesNoCheck()) {
                        if (rpABytesNoCheckEntry != null)
                            output.writeByteArray(FIELD_RP_ABYTES_NO_CHECK, rpABytesNoCheckEntry, false);
                    }
                }
                break;
            case FIELD_RP_BYTES:
                if (message.getRpBytes() != null) {
                    for (ByteString rpBytesEntry : message.getRpBytes()) {
                        if (rpBytesEntry != null)
                            output.writeBytes(FIELD_RP_BYTES, rpBytesEntry, true);
                    }
                }
                break;

Original comment by david.yu...@gmail.com on 29 Mar 2012 at 3:05

[GoogleCodeExporter]

No. In our project we are using only byte[] fields. But this is there for 
compatibility.

Idea is the following:
1) by default "bytes" types do writes and reads compatible with java_bean - 
ByteString.
2) if you specify @Bytes annotation on a bytes field it will generat 
readByteArray/writeByteArray for the field for you (so this is byte[] case)
3) if you want for some special cases, you can use @NoCheck annotation, to ask 
to remove check/creation of collection for the field in mergeFrom switch case. 
(i.e.
if (message.getRpBytes() == null)
    message.setRpBytes(new ArrayList<ByteString>());)

so with @NoCheck you will have
case FIELD_RP_ABYTES_NO_CHECK:
    message.getRpABytesNoCheck().add(input.readByteArray());
    break;
instead of 
case FIELD_RP_ABYTES_NO_CHECK:
    if (message.getRpABytesNoCheck() == null)
        message.setRpABytesNoCheck(new ArrayList<byte[]>());
    message.getRpABytesNoCheck().add(input.readByteArray());
    break;

Original comment by John.Ko...@gmail.com on 29 Mar 2012 at 3:13

[GoogleCodeExporter]

It will be best that the changes you add will be somewhat like a mirror to the 
changes on your build.  E.g no extra stuff.

If you prefer not to have a custom build and include your changes directly to 
protostuff, that would be fine as well.  I can add you to the project.

If you prefer the latter, let me know what account you will use for svn write 
access.

Original comment by david.yu...@gmail.com on 29 Mar 2012 at 6:27

[GoogleCodeExporter]

But if you are committed to maintain it via patches, I'm also fine with that.

I'm particular about this thing because it happened before with the j2me module 
(which I do not use) ... and I was the one ending up having to clean up the 
mess of the contributor because he was not committed.  I had to do it rather 
than shipping unmaintained software full of bugs.

So if you feel a github project is better for you, I'm fine with that as well.

Original comment by david.yu...@gmail.com on 30 Mar 2012 at 5:00

[GoogleCodeExporter]

I don't mind to commit changes to this compiler directly to 
protostuff-compiler. 
But the point is that i can't do full support 24x7 hours a week for this). This 
compilers were developed for our project to maintain inheritance and 
serializations problems in our models. So it works and it meets our 
requirements. And sadly saing i have no time to do some extra stuff beyond 
this. So it's up for you to decide what it will be better for your project. I 
can't promise you that i will be able to support this next month/year. From 
this point of view may be it will be better if i just commit it to some place 
at github.

I just thought this code can be useful for someone. ). I understand of course 
that without support it's not as good as it might be. but that's all i've got.

Original comment by John.Ko...@gmail.com on 30 Mar 2012 at 9:15

[GoogleCodeExporter]

I don't mind if you can't commit as long as there are no more lurking bugs ... 
like the one I discovered :-)

What I usually do is I declare all possible field types with all 3 types 
(optional, required, repeated), generate the output, and then scan them one by 
one if there are bugs.

I'm not sure if there is an automated way to test templating code.

Would it be better if I include this on the next release instead? (or do you 
think it is good enough to be included in 1.0.5)?
Either way, you will still need to document how to use it.

If on the next release, I think I'll go ahead and do 1.0.5 today.

Original comment by david.yu...@gmail.com on 30 Mar 2012 at 11:48

[GoogleCodeExporter]

ok, let's include it on the next release cause i should anyway to document it 
and fix mentioned bug.

Original comment by John.Ko...@gmail.com on 30 Mar 2012 at 11:53

[GoogleCodeExporter]

so, how should i document it?

Original comment by John.Ko...@gmail.com on 31 Mar 2012 at 3:00

[GoogleCodeExporter]

You can create a new wiki entry (and reference it from 
http://code.google.com/p/protostuff/wiki/CompilerOptions).  

You now have write access.
With regards to formatting, only 2 things:
1. spaces over tabs
2. curly brackets on a new line

Thanks :-) 

Original comment by david.yu...@gmail.com on 31 Mar 2012 at 3:50

[GoogleCodeExporter]

I just noticed the wiki updates.  Its a good start.
Note that for protos, it is ok to have curly braces on same line (like the 
existing examples).  What I mentioned before applies to java sources only.

Thanks for contributing!

Original comment by david.yu...@gmail.com on 10 Apr 2012 at 12:41

[GoogleCodeExporter]

It seems there are only great samples left to be provided in the doc. Check it 
out. Is it good enought or i should fix something there?

Original comment by John.Ko...@gmail.com on 14 Apr 2012 at 4:38

[GoogleCodeExporter]

Include the generated output so that they'll have an idea what it looks like.
Apart from that, I think its good.  Only thing left is committing the actual 
stg that generates those output.

Original comment by david.yu...@gmail.com on 17 Apr 2012 at 12:16

[GoogleCodeExporter]

Ok, i will prepare better example to show inheritance and overriding. And 
result of generation. 

There also may be some more options will be available for this schema 
generator. 
I would like to add to this generator ability to generate 
mergeFrom(java.sql.ResultSet) also. And it seems to be we will need to add 
deserialization ability using mergeFrom(String[]). (to support HandlerSocket 
from MySQL too). May be this generator is not best place to include this 
functionality inside. But it would be nice, to have those deser. supports in 
one schema if needed. Don't you mind? (may be it should be placed somewhere 
else)

Original comment by John.Ko...@gmail.com on 17 Apr 2012 at 12:29

[GoogleCodeExporter]

I don't mind at all.  Actually I'm interested to see how you've integrated it 
with sql and handlersocket.  I myself use protostuff with leveldb (custom 
generators too, its too specific to the project though).

Original comment by david.yu...@gmail.com on 17 Apr 2012 at 12:32

[GoogleCodeExporter]

I have updated the page with advanced inheritance in collections sample.

Original comment by John.Ko...@gmail.com on 23 Apr 2012 at 9:42

[GoogleCodeExporter]

This generator has a great usecase for people who have existing pojos, and want 
to avoid writing boilerplate IO code.  Its flexibility (inheritable/can be 
overridden) is definitely a plus.  Nice work!

Original comment by david.yu...@gmail.com on 24 Apr 2012 at 7:22

[GoogleCodeExporter]

Ok )) how should i bring the sources up to you? put them to svn?

Original comment by John.Ko...@gmail.com on 24 Apr 2012 at 7:25

[GoogleCodeExporter]

Yea svn (trunk)

Original comment by david.yu...@gmail.com on 24 Apr 2012 at 7:42

[GoogleCodeExporter]

Tried to commit, but can't build last sources 1.0.6-SNAPSHOT from scratch. 
(test failed in runtime-md). r1504.

Tests in error: 
  testPojoWithClassFields(com.dyuproject.protostuff.runtime.ProtobufRuntimeObjectSchemaTest)
  testPojoWithClassFields(com.dyuproject.protostuff.runtime.ProtostuffRuntimeObjectSchemaTest)

Original comment by John.Ko...@gmail.com on 28 Apr 2012 at 1:18

[GoogleCodeExporter]

Try again.  I just committed fix.
Thanks!

Original comment by david.yu...@gmail.com on 28 Apr 2012 at 1:55

[GoogleCodeExporter]

Ok. 

There is one more question.

In my own copy of protostuff i had modified the root Compiler class so my 2 
compilers to be included directly as your others. (...add(new MyCompiler()); 
... )

Should i commit only source code of my 2 new compilers to protostuff-compiler, 
or modified main Compiler class too?

Original comment by John.Ko...@gmail.com on 28 Apr 2012 at 2:34

[GoogleCodeExporter]

Feel free to include it in the root compiler.  So yes, you can commit the 
modified Compiler class too.

Original comment by david.yu...@gmail.com on 28 Apr 2012 at 2:40

[GoogleCodeExporter]

I have commited the sources. Now this is inside of your project ;)
Have fixed brackets, and repeated boolean flag on repeated fields.

Original comment by John.Ko...@gmail.com on 29 Apr 2012 at 11:12

[GoogleCodeExporter]

just thought this should be added to VERSION.txt for 1.0.6-SNAPSHOT changes. 
you do that for yourself?

Original comment by John.Ko...@gmail.com on 29 Apr 2012 at 11:37

[GoogleCodeExporter]

I've updated version.txt.  
Also, feel free to update 
http://code.google.com/p/protostuff/wiki/CompilerOptions and include your 
options there as well.
Thanks!

Original comment by david.yu...@gmail.com on 30 Apr 2012 at 2:37

• Changed title: 2 new compilers: proto_extender, java_bean_model
• Changed state: Fixed