Annotation extension for ASN library

[GoogleCodeExporter]

Add annotation extension for ASN library

Original issue reported on code.google.com by amit.bha...@gmail.com on 24 Jun 2012 at 8:22

[GoogleCodeExporter]

Hi Amit,

I've finished the encoder part of the annotation extension for your ASN.1 
library. Still didn't have a chance to properly test it, so there is likely a 
number of bugs still in the code, but I wanted to show you how it's coming 
along. I've also put together a simple performance test tool that compares your 
old encoding to mine. The current test case it uses is quite simple (a SEQUENCE 
with an OCTET-STRING and a NULL value), but these were the only common 
structures I could find between what's already in the MAP stack and the stuff 
we've implemented for UPDATE_GPRS_LOCATION. We can compare more complex 
structures later, but I don't expect those to perform any different, since the 
structure of the code is essentially the same.

Attached in this e-mail is a patch for the ASN.1 library to extend it with the 
annotation parser and the source of my performance test tool. I've compiled a 
single jar binary, that you can run, but it's 8.8 MB, so you can download it 
from here: 
http://static.bodytrace.com/gsm/AsnPerfTest-1.0-SNAPSHOT-jar-with-dependencies.j
ar
The output of an execution is at the bottom of my e-mail. As you can see the 
performance is exactly the same as with your code.

I've used some Java 7 specific things, so source and target values in the 
pom.xml of the asn root need to be changed from 1.5 to 1.7, but these could be 
coded around later.

Cheers,
Andy

---
Initializing...done

Outputs: 
Old: 30 0C 80 08 21 43 65 87 09 21 43 65 81 00 
New: 30 0C 80 08 21 43 65 87 09 21 43 65 81 00 

Warming up...done
Executing performance test (results in milliseconds):
Count   Old New
----------------------
0:  224 199
1:  221 200
2:  222 200
3:  221 199
4:  224 199
5:  222 202
6:  225 199
7:  224 199
8:  224 201
9:  223 199
10: 222 199
11: 233 199
12: 223 208
13: 230 201
14: 225 200
15: 229 202
16: 224 198
17: 227 200
18: 225 200
19: 223 200
20: 227 199
21: 227 224
22: 226 201
23: 224 200
24: 223 200
25: 224 199
26: 224 201
27: 224 200
28: 223 201
29: 221 201
30: 224 200
31: 224 202
32: 227 199
33: 223 201
34: 224 202
35: 224 200
36: 225 200
37: 224 201
38: 221 200
39: 223 201
40: 226 199
41: 221 199
42: 223 197
43: 225 201
44: 224 199
45: 223 201
46: 225 201
47: 224 201
48: 226 201
49: 224 201
50: 226 202
51: 228 199
52: 225 201
53: 230 202
54: 225 200
55: 226 203
56: 224 200
57: 223 201
58: 225 201
59: 224 199
60: 223 198
61: 223 200
62: 225 201
63: 224 199
64: 224 231
65: 224 200
66: 227 203
67: 226 200
68: 224 199
69: 225 204
70: 224 201
71: 225 200
72: 228 200
73: 224 202
74: 236 201
75: 225 201
76: 225 201
77: 226 201
78: 225 201
79: 225 200
80: 222 200
81: 223 200
82: 223 200
83: 225 202
84: 227 200
85: 225 201
86: 227 199
87: 227 201
88: 226 200
89: 224 201
90: 225 202
91: 227 202
92: 224 201
93: 225 199
94: 224 200
95: 224 223
96: 224 199
97: 226 201
98: 224 200
99: 223 199
----------------------
AVG:    224 201

Original comment by amit.bha...@gmail.com on 24 Jun 2012 at 8:23

Attachments:

• asn-annotations-encoder.diff
• AsnPerfTest.zip

[GoogleCodeExporter]

Just finished the code clean-up. This patch now has both the encoder and 
decoder and some unit tests as well. Please review the code and functionality, 
but it should be relatively stable now. I had to do a major refactoring, 
because having everything in a single method slowed things down considerable as 
the method body grew larger. I've now split every object into its own 
encoder/decoder method pair, this should work well.

I've also updated the MAP objects in the performance tool, they now have the 
final annotations and a simple ExtensionContainer has been added as well. Both 
source and runnable JAR file are attached.

A few remarks:
- the code still uses Java 7 specific syntax, if this is a problem, let me 
know; otherwise source and target values in the pom.xml of the asn root need to 
be changed from 1.5 to 1.7
- ENUMERATED types are decoded strictly, if a value in the ASN.1 input is not 
recognized, an exception is thrown
- CHOICE decoding throws an exception if no field is recognized
- SEQUENCE decoding is quite relaxed, if an unexpected tag is encountered, it 
is simply skipped, the order of the fields (assigned by the position parameter) 
is enforced, however
- fields with type byte[] are encoded as is, and decoded by copying the 
contests of the element into the array; they are always encoded and expected to 
be coded as non-primitive and must always have a context sensitive tag value 
defined
- the unit tests are not from packet traces, they are mostly self-generated or 
generated using another ASN.1 parser, please review these carefully to make 
sure that they are indeed valid
- the @Range annotation is not checked at all at the moment

Finally, I'd like to recommend that you change the bundled JUnit to a more 
recent version that supports the assertArrayEquals() method to make unit 
testing with arrays more user-friendly.

These are the latest benchmarks I've made. It did a 1,000,000 cycles repeated 
100 times the results of which were then averaged. It's still using the simple 
ADDInfo object, it'd be interesting to see how the new code performs with 
bigger structures.

Process         Old vs New (msec)
--------------------------------------
Encode only:        235 vs 149 (63 % faster)
Decode only:        177 vs 102 (57 % faster)
Encode & decode:    386 vs 237 (61 % faster)

Original comment by a...@bodytrace.com on 27 Jun 2012 at 8:32

Attachments:

• mobicents-asn-20120627.patch
• AsnPerfTest-20120627.zip
• AsnPerfTest-20120627.jar

[GoogleCodeExporter]

Attached is the latest version of the annotation based ASN.1 coder. It adds the 
capability to encode and decode annotated POJOs without having to write 
encoder/decoder code by hand. It does this by searching for annotated objects 
on startup, examining them with reflection and generating code to encode and 
decode these objects on the fly. This only happens once at startup, after that 
reflection is not used. The resulting code generated performs just the same as 
the current hard-coded code in the protocol stack.

Dynamic code generation is performed based on templates in 
src/main/resources/org/mobicents/protocols/asn/template. Additional types can 
be easily added by creating a template and updating the dispatcher functions.

Current functionality of jasn is not impacted at all, this is just an addition, 
so code that uses jasn needn't be changed, however new features/objects could 
be implemented using annotations instead.

The dynamic coder currently supports the following types: BIT STRING, CHOICE, 
ENUMERATED, OBJECT IDENTIFIER, OCTET STRING, SEQUENCE, and SEQUENCE OF. These 
seem to be enough for MAP functionality.

Since the last patch, I've done some major refactoring and added a large number 
of unit tests which have been verified against a third-party ASN.1 tool.

A simple annotated SEQUENCE object looks like this:
---
@Sequence("Sequence-type3")
public class SequenceTest3 {

    @OrderedField(name="choice-test", position=0) ChoiceTest choiceValue;
    @OrderedField(name="null-test", position=1) @Optional boolean nullValue;

    ... (constructor, getters/setters) ...
}
---

With the optional use of Project Lombok, the entire class definition can be 
reduced to just a few lines of field declarations, everything else, including 
constructors, getters/setters, hashCode/equals methods can then be generated 
automatically, which results is very light-weight and easy to understand code.

To encode an object, the coder first needs to be initialized at startup:
---
Coder.init("package.holding.all.annotated.objects");
---

After which encoding is as simple as:
---
SequenceTest3 s = new SequenceTest3();
...
AsnOutputStream asnOs = new AsnOutputStream();
asnOs.writeObject(s);
---

To decode the object, the following is needed:
---
AsnInputStream asnIs = new AsnInputStream();
...
SequenceTest3 s = asnIs.readObject(SequenceTest3.class);
---

Encoding SEQUENCE OF types is also very straightforward, they just need to be 
passed in as Component[] arrays, where Component is the class of the object the 
SEQUENCE OF array will hold (e.g.. SequenceTest3[]).

More examples can be found in the unit tests, but operation is pretty self 
explanatory. 

Using this coder could greatly reduce the work needed to implement new 
functionality and/or messages in the protocol stack. I'm looking at 
implementing an ASN.1 parser that would generate annotated objects 
automatically. Using that, it would be fairly simple to extend the current MAP 
stack to support pretty much all operations and could potentially be used for 
CAP and other protocols as well.

I have attached a diff that can be applied against the current jasn code in 
jasn/asn-impl. It does the following:
- adds annotation classes in org.mobicents.protocols.asn.annotation package
- adds necessary interfaces & ready to use classes for OBJECT IDENTIFIER and 
OCTET STRING in org.mobicents.protocols.asn.type
- adds Coder class to manage the coder and the DynamicCoder interface for 
internal use to org.mobicents.protocols.asn
- adds setup and encoder/decoder methods to AsnInputStream and AsnOutputStream 
classes
- adds templates to src/main/resources/org/mobicents/protocols/asn/template
- adds unit tests to AsnInputStreamTest and AsnOutputStreamTest
- adds classes used by unit tests in org.mobicents.protocols.asn.object test 
package
- adds dependencies to pom.xml for javassist (dynamic code generation), 
reflections (reflection helper) and freemarker (template engine)

I've modified the code, so it should compile fine with Java 5.

Original comment by a...@bodytrace.com on 10 Apr 2013 at 11:48

Attachments:

• jasn-dynamiccoder.patch