search

tcalmant / python-javaobj

Extended fork of python-javaobj from http://code.google.com/p/python-javaobj/
Apache License 2.0
77 stars 19 forks source link

Handle the SC_WRITE_METHOD class description flag #1

Open noragen opened 9 years ago

noragen commented 9 years ago 
>>> val
'\xac\xed\x00\x05sr\x00-net.centersight.model.datapoint.DatapointBean\x00\x00\x00\x00\x00\x00\x00\x01\x02\x00\x11L\x00\x06accesst\x00\x13Ljava/lang/Integer;L\x00\x07addresst\x00\x12Ljava/lang/String;L\x00\x08categoryq\x00~\x00\x01L\x00\x0fcounterTimeBaseq\x00~\x00\x01L\x00\x08dataTypeq\x00~\x00\x01L\x00\rdatapointTypeq\x00~\x00\x01L\x00\x0cdisplayLevelq\x00~\x00\x01L\x00\nidentifiert\x00\x10Ljava/lang/Long;L\x00\tisCountert\x00\x13Ljava/lang/Boolean;L\x00\nlastUpdatet\x00\x10Ljava/util/Date;L\x00\x0clastValueStrq\x00~\x00\x02L\x00\rprevLastValuet\x004Lnet/centersight/model/datapoint/DatapointValueBean;L\x00\x10prevLastValueStrq\x00~\x00\x02L\x00\x0bstorageTypeq\x00~\x00\x01L\x00\x10transformDisplayq\x00~\x00\x02L\x00\x0btransformInq\x00~\x00\x02L\x00\x04unitq\x00~\x00\x02xr\x00\x1enet.centersight.model.NodeBean\x00\x00\x00\x00\x00\x00\x00\x01\x02\x00\x12I\x00\x05levelI\x00\x08nodeTypeI\x00\tsortIndexL\x00\ralarmSeverityq\x00~\x00\x01L\x00\x10customNodeTypeIdq\x00~\x00\x03L\x00\x0bdescriptionq\x00~\x00\x02L\x00\x02idq\x00~\x00\x03L\x00\x0blastUpdatedq\x00~\x00\x05L\x00\rlastUpdatedByq\x00~\x00\x02L\x00\nlinkedNodet\x00 Lnet/centersight/model/NodeBean;L\x00\x0bmapLatitudet\x00\x12Ljava/lang/Double;L\x00\x0cmapLongitudeq\x00~\x00\tL\x00\x07mapZoomq\x00~\x00\x01L\x00\x04nameq\x00~\x00\x02L\x00\x08parentIdq\x00~\x00\x03L\x00\x04pathq\x00~\x00\x02L\x00\x08subNodest\x00\x10Ljava/util/List;L\x00\x08timeZonet\x00\x14Ljava/util/TimeZone;xp\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00pppsr\x00\x0ejava.lang.Long;\x8b\xe4\x90\xcc\x8f#\xdf\x02\x00\x01J\x00\x05valuexr\x00\x10java.lang.Number\x86\xac\x95\x1d\x0b\x94\xe0\x8b\x02\x00\x00xp\x00\x00\x00\x00\x007Oxsr\x00\x12java.sql.Timestamp&\x18\xd5\xc8\x01S\xbfe\x02\x00\x01I\x00\x05nanosxr\x00\x0ejava.util.Datehj\x81\x01KYt\x19\x03\x00\x00xpw\x08\x00\x00\x01I\xcd`\xfdpx!\xf9\x82\x80t\x00\x06_ROOT_ppppt\x00\rMESSAGE_ERRORsq\x00~\x00\r\x00\x00\x00\x00\x00+\xb3(t\x00\x1b625/2863909/2863912/3624824ppppppsr\x00\x11java.lang.Integer\x12\xe2\xa0\xa4\xf7\x81\x878\x02\x00\x01I\x00\x05valuexq\x00~\x00\x0e\x00\x00\x00\x04sq\x00~\x00\x17\x00\x00\x00\nppsr\x00\x11java.lang.Boolean\xcd r\x80\xd5\x9c\xfa\xee\x02\x00\x01Z\x00\x05valuexp\x00ppppppt\x00\rMESSAGE_ERRORp'

>>> val_deserilized=javaobj.loads(val)
No handlers could be found for logger "javaobj"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/javaobj.py", line 180, in loads
    return marshaller.readObject()
  File "/usr/local/lib/python2.6/dist-packages/javaobj.py", line 383, in readObject
    _, res = self._read_and_exec_opcode(ident=0)
  File "/usr/local/lib/python2.6/dist-packages/javaobj.py", line 448, in _read_and_exec_opcode
    return (opid, handler(ident=ident))
  File "/usr/local/lib/python2.6/dist-packages/javaobj.py", line 674, in do_object
    res = self._read_value(field_type, ident, name=field_name)
  File "/usr/local/lib/python2.6/dist-packages/javaobj.py", line 862, in _read_value
    _, res = self._read_and_exec_opcode(ident=ident + 1)
  File "/usr/local/lib/python2.6/dist-packages/javaobj.py", line 446, in _read_and_exec_opcode
    .format(opid))
RuntimeError: Unknown OpCode in the stream: 0x1
tcalmant commented 9 years ago

I've just made a small test and reproduced this error.

==Oops state dump=============
References: [[net.centersight.model.datapoint.DatapointBean:0x1],
    'Ljava/lang/Integer;', 'Ljava/lang/String;', 'Ljava/lang/Long;',
    'Ljava/lang/Boolean;', 'Ljava/util/Date;',
    'Lnet/centersight/model/datapoint/DatapointValueBean;',
    [net.centersight.model.NodeBean:0x1], 'Lnet/centersight/model/NodeBean;',
    'Ljava/lang/Double;', 'Ljava/util/List;', 'Ljava/util/TimeZone;',
    <javaobj:net.centersight.model.datapoint.DatapointBean>,
    [java.lang.Long:0x3B8BE490CC8F23DF], [java.lang.Number:0x-79536AE2F46B1F75],
    <javaobj:java.lang.Long>, [java.sql.Timestamp:0x2618D5C80153BF65],
    [java.util.Date:0x686A81014B597419], <javaobj:java.sql.Timestamp>]
Stream seeking back at -16 byte (2nd line is an actual position!):
Warning!!!!: Stream still has 233 bytes left.
0000   81 01 4B 59 74 19 03 00 00 78 70 77 08 00 00 01   ..KYt....xpw....
0010   49 CD 60 FD 70 78 21 F9 82 80 74 00 06 5F 52 4F   IÍ`ýpx!ù..t.._RO
0020   4F 54 5F 70 70 70 70 74 00 0D 4D 45 53 53 41 47   OT_ppppt..MESSAG
0030   45 5F 45 52 52 4F 52 73 71 00 7E 00 0D 00 00 00   E_ERRORsq.~.....
0040   00 00 2B B3 28 74 00 1B 36 32 35 2F 32 38 36 33   ..+³(t..625/2863
0050   39 30 39 2F 32 38 36 33 39 31 32 2F 33 36 32 34   909/2863912/3624
0060   38 32 34 70 70 70 70 70 70 73 72 00 11 6A 61 76   824ppppppsr..jav
0070   61 2E 6C 61 6E 67 2E 49 6E 74 65 67 65 72 12 E2   a.lang.Integer.â
0080   A0 A4 F7 81 87 38 02 00 01 49 00 05 76 61 6C 75   .¤÷..8...I..valu
0090   65 78 71 00 7E 00 0E 00 00 00 04 73 71 00 7E 00   exq.~......sq.~.
00A0   17 00 00 00 0A 70 70 73 72 00 11 6A 61 76 61 2E   .....ppsr..java.
00B0   6C 61 6E 67 2E 42 6F 6F 6C 65 61 6E CD 20 72 80   lang.BooleanÍ r.
00C0   D5 9C FA EE 02 00 01 5A 00 05 76 61 6C 75 65 78   Õ.úî...Z..valuex
00D0   70 00 70 70 70 70 70 70 74 00 0D 4D 45 53 53 41   p.ppppppt..MESSA
00E0   47 45 5F 45 52 52 4F 52 70                        GE_ERRORp

==============================
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\javaobj.py", line 443, in _read_and_exec_opcode
    handler = self.opmap[opid]
KeyError: 1

#... (after that, its the same trace as yours)

I can't work on it today, but I'll take a look when I'll be able.

noragen commented 9 years ago

Great! :)

PS.: Don't forget ;)

tcalmant commented 9 years ago

Just a comment to tell you I haven't forget about this issue ;) So far, I've found that this might be due to the way java.sql.Timestamp is serialized. Its parent class java.util.Date has a customized serialization: it just writes an integer timestamp instead of associating its fields with a value. This seems to cause an inconsistent reading of the class by Javaobj and to miss the next object, and to raise the error you see. In Java, this is not problem, as the class of the loaded object is available and the reader can call the Date.readObject(InputObjectStream) method.

I'm still looking for a workaround for javaobj :/

j8spencer commented 6 years ago

I believe the same thing is happening for ImmutableListMultiMap (maybe just the version from Guava.) The serializer seems to write the number of key/value sets as block data, then for each set, writes: 1.) the key, 2.) blockdata with the count of the values, 3.) the list of values. In most cases, this seems to get sucked into the annotations and at least the stream is consistent (though the keys/values are not accessible in any convenient way.) But for an EmptyImmuntableListMultimap, the block data saying there are 0 k/v sets becomes the value for the following object.

I think one thing that might help is checking not just the classdesc flags, but the collapsed flags for all superclasses, for SC_WRITE_METHOD. It seems any time a parent class uses SC_WRITE_METHOD, there's a chance (nay, a guarantee) that one part of the object has used a blockdata--though it may be in the middle of a three-class hierarchy! The current code doesn't seem to OR the flags together, but instead uses the values from the lowest level class. It's also not clear how to interpret this when a class is referenced and that class has SC_WRITE_METHOD, or how to interpret it in the context of an object (where the classdec has already been read--it seems that it's being used for object data, not one-time class description data.)

urucoder commented 4 years ago

This case is readable by the version 2 of the package, should this issue be closed?

j8spencer commented 4 years ago

I believe I had the issue in version 2. Can you give me the commit id you used for testing and I'll re-check?

urucoder commented 4 years ago

I tested the @noragen binary with the current master state and it works.

`import javaobj.v2 as javaobj

test = b'\xac\xed\x00\x05sr\x00-net.centersight.model.datapoint.DatapointBean\x00\x00\x00\x00\x00\x00\x00\x01\x02\x00\x11L\x00\x06accesst\x00\x13Ljava/lang/Integer;L\x00\x07addresst\x00\x12Ljava/lang/String;L\x00\x08categoryq\x00~\x00\x01L\x00\x0fcounterTimeBaseq\x00~\x00\x01L\x00\x08dataTypeq\x00~\x00\x01L\x00\rdatapointTypeq\x00~\x00\x01L\x00\x0cdisplayLevelq\x00~\x00\x01L\x00\nidentifiert\x00\x10Ljava/lang/Long;L\x00\tisCountert\x00\x13Ljava/lang/Boolean;L\x00\nlastUpdatet\x00\x10Ljava/util/Date;L\x00\x0clastValueStrq\x00~\x00\x02L\x00\rprevLastValuet\x004Lnet/centersight/model/datapoint/DatapointValueBean;L\x00\x10prevLastValueStrq\x00~\x00\x02L\x00\x0bstorageTypeq\x00~\x00\x01L\x00\x10transformDisplayq\x00~\x00\x02L\x00\x0btransformInq\x00~\x00\x02L\x00\x04unitq\x00~\x00\x02xr\x00\x1enet.centersight.model.NodeBean\x00\x00\x00\x00\x00\x00\x00\x01\x02\x00\x12I\x00\x05levelI\x00\x08nodeTypeI\x00\tsortIndexL\x00\ralarmSeverityq\x00~\x00\x01L\x00\x10customNodeTypeIdq\x00~\x00\x03L\x00\x0bdescriptionq\x00~\x00\x02L\x00\x02idq\x00~\x00\x03L\x00\x0blastUpdatedq\x00~\x00\x05L\x00\rlastUpdatedByq\x00~\x00\x02L\x00\nlinkedNodet\x00 Lnet/centersight/model/NodeBean;L\x00\x0bmapLatitudet\x00\x12Ljava/lang/Double;L\x00\x0cmapLongitudeq\x00~\x00\tL\x00\x07mapZoomq\x00~\x00\x01L\x00\x04nameq\x00~\x00\x02L\x00\x08parentIdq\x00~\x00\x03L\x00\x04pathq\x00~\x00\x02L\x00\x08subNodest\x00\x10Ljava/util/List;L\x00\x08timeZonet\x00\x14Ljava/util/TimeZone;xp\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00pppsr\x00\x0ejava.lang.Long;\x8b\xe4\x90\xcc\x8f#\xdf\x02\x00\x01J\x00\x05valuexr\x00\x10java.lang.Number\x86\xac\x95\x1d\x0b\x94\xe0\x8b\x02\x00\x00xp\x00\x00\x00\x00\x007Oxsr\x00\x12java.sql.Timestamp&\x18\xd5\xc8\x01S\xbfe\x02\x00\x01I\x00\x05nanosxr\x00\x0ejava.util.Datehj\x81\x01KYt\x19\x03\x00\x00xpw\x08\x00\x00\x01I\xcd\xfdpx!\xf9\x82\x80t\x00\x06_ROOT_ppppt\x00\rMESSAGE_ERRORsq\x00~\x00\r\x00\x00\x00\x00\x00+\xb3(t\x00\x1b625/2863909/2863912/3624824ppppppsr\x00\x11java.lang.Integer\x12\xe2\xa0\xa4\xf7\x81\x878\x02\x00\x01I\x00\x05valuexq\x00~\x00\x0e\x00\x00\x00\x04sq\x00~\x00\x17\x00\x00\x00\nppsr\x00\x11java.lang.Boolean\xcd r\x80\xd5\x9c\xfa\xee\x02\x00\x01Z\x00\x05valuexp\x00ppppppt\x00\rMESSAGE_ERRORp' pobj = javaobj.loads(test)