fix json decode error

[vangheem]

args is not always a single object from datetime.datetime(dt_bytes(*args)).isoformat() for example

[jimfulton]

Thanks! Test case? Example data that illustrates the problem?

[vangheem]

sorry, I tried but I didn't have enough time to figure out how to load my pickle that causes the error with the JsonUnpickler correctly(or a fake class that produces the problem).

The problem is in the REDUCE method. Seems f, args = self.pop(2) produces an args like this:

(<newt.db.jsonpickle.Bytes object at 0x10947db38>, <newt.db.jsonpickle.Put object at 0x10947d400>)

which is passed to the instance function which applies those args to the special_classes instance and expects only 1 argument.

[jimfulton]

No problem. If you can share a pickle that provokes this, I'll run it down.

I predict a datetime with a time zone.

[vangheem]

Sure, it's a class instance from plone.server.content.Folder:

b'\x80\x03}q\x01(X\x08\x00\x00\x00__name__q\x02X\x0b\x00\x00\x00somefolder4q\x03X\x04\x00\x00\x00uuidq\x04X \x00\x00\x0029c497efdd8749d48df7ecba969cb490q\x05X\x0c\x00\x00\x00__provides__q\x06czope.interface.declarations\nProvides\nq\x07h\x00cplone.server.behaviors.dublincore\nIMarkerDublinCore\nq\x08\x86q\tRq\nX\x08\x00\x00\x00creatorsq\x0bX\x04\x00\x00\x00rootq\x0c\x85q\rX\x11\x00\x00\x00modification_dateq\x0ecdatetime\ndatetime\nq\x0fC\n\x07\xe1\x01\x1d\x14\x180\x02_\x98q\x10cdateutil.tz.tz\ntzlocal\nq\x11)\x81q\x12}q\x13(X\x0b\x00\x00\x00_std_offsetq\x14cdatetime\ntimedelta\nq\x15J\xff\xff\xff\xffM \xfdK\x00\x87q\x16Rq\x17X\x0b\x00\x00\x00_dst_offsetq\x18h\x15J\xff\xff\xff\xffJ0\x0b\x01\x00K\x00\x87q\x19Rq\x1aX\x07\x00\x00\x00_hasdstq\x1b\x88X\n\x00\x00\x00_dst_savedq\x1ch\x15K\x00M\x10\x0eK\x00\x87q\x1dRq\x1eub\x86q\x1fRq X\x0b\x00\x00\x00portal_typeq!X\x06\x00\x00\x00Folderq"X\r\x00\x00\x00creation_dateq#h X\x05\x00\x00\x00titleq$X\r\x00\x00\x00Some folder 4q%X\r\x00\x00\x00_Folder__dataq&C\x08\x00\x00\x00\x00\x00\x00\x00\xd2q\'cBTrees.OOBTree\nOOBTree\nq(\x86q)QX\x0c\x00\x00\x00_Folder__lenq*C\x08\x00\x00\x00\x00\x00\x00\x00\xd3q+cBTrees.Length\nLength\nq,\x86q-QX\n\x00\x00\x00__parent__q.C\x08\x00\x00\x00\x00\x00\x00\x00aq/cplone.server.content\nSite\nq0\x86q1QX\x0c\x00\x00\x00contributorsq2h\x0c\x85q3X\x0f\x00\x00\x00__annotations__q4C\x08\x00\x00\x00\x00\x00\x00\x00\xd4q5h(\x86q6Qu.'

[jimfulton]

That looks like that is the second part, state, of a database record. Can you share the entire record please?

[vangheem]

You mean the record in postgres?

[jimfulton]

I mean the value of object_state.state

[vangheem]

select state from object_state where zoid=209;

\x800363706c6f6e652e7365727665722e636f6e74656e740a466f6c6465720a71002e80037d71012858080000005f5f6e616d655f5f7102580b000000736f6d65666f6c6465723471035804000000757569647104582000000032396334393765666464383734396434386466376563626139363963623439307105580c0000005f5f70726f76696465735f5f7106637a6f70652e696e746572666163652e6465636c61726174696f6e730a50726f76696465730a7107680063706c6f6e652e7365727665722e6265686176696f72732e6475626c696e636f72650a494d61726b65724475626c696e436f72650a710886710952710a580800000063726561746f7273710b5804000000726f6f74710c85710d58110000006d6f64696669636174696f6e5f64617465710e636461746574696d650a6461746574696d650a710f430a07e1011d141830025f98711063646174657574696c2e747a2e747a0a747a6c6f63616c0a7111298171127d711328580b0000005f7374645f6f66667365747114636461746574696d650a74696d6564656c74610a71154affffffff4d20fd4b00877116527117580b0000005f6473745f6f6666736574711868154affffffff4a300b01004b0087711952711a58070000005f686173647374711b88580a0000005f6473745f7361766564711c68154b004d100e4b0087711d52711e756286711f527120580b000000706f7274616c5f7479706571215806000000466f6c6465727122580d0000006372656174696f6e5f646174657123682058050000007469746c657124580d000000536f6d6520666f6c64657220347125580d0000005f466f6c6465725f5f646174617126430800000000000000d27127634254726565732e4f4f42547265650a4f4f42547265650a712886712951580c0000005f466f6c6465725f5f6c656e712a430800000000000000d3712b634254726565732e4c656e6774680a4c656e6774680a712c86712d51580a0000005f5f706172656e745f5f712e43080000000000000061712f63706c6f6e652e7365727665722e636f6e74656e740a536974650a713086713151580c000000636f6e7472696275746f72737132680c857133580f0000005f5f616e6e6f746174696f6e735f5f7134430800000000000000d47135682886713651752e

Full record:

zodb=# select * from object_state where zoid=209;
 zoid |        tid         | prev_tid |               md5                | state_size |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 state                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
------+--------------------+----------+----------------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  209 | 269365797257436023 |        0 | 754a71e17bdb5f32b3555957eb1f7afb |        850 | \x800363706c6f6e652e7365727665722e636f6e74656e740a466f6c6465720a71002e80037d71012858080000005f5f6e616d655f5f7102580b000000736f6d65666f6c6465723471035804000000757569647104582000000032396334393765666464383734396434386466376563626139363963623439307105580c0000005f5f70726f76696465735f5f7106637a6f70652e696e746572666163652e6465636c61726174696f6e730a50726f76696465730a7107680063706c6f6e652e7365727665722e6265686176696f72732e6475626c696e636f72650a494d61726b65724475626c696e436f72650a710886710952710a580800000063726561746f7273710b5804000000726f6f74710c85710d58110000006d6f64696669636174696f6e5f64617465710e636461746574696d650a6461746574696d650a710f430a07e1011d141830025f98711063646174657574696c2e747a2e747a0a747a6c6f63616c0a7111298171127d711328580b0000005f7374645f6f66667365747114636461746574696d650a74696d6564656c74610a71154affffffff4d20fd4b00877116527117580b0000005f6473745f6f6666736574711868154affffffff4a300b01004b0087711952711a58070000005f686173647374711b88580a0000005f6473745f7361766564711c68154b004d100e4b0087711d52711e756286711f527120580b000000706f7274616c5f7479706571215806000000466f6c6465727122580d0000006372656174696f6e5f646174657123682058050000007469746c657124580d000000536f6d6520666f6c64657220347125580d0000005f466f6c6465725f5f646174617126430800000000000000d27127634254726565732e4f4f42547265650a4f4f42547265650a712886712951580c0000005f466f6c6465725f5f6c656e712a430800000000000000d3712b634254726565732e4c656e6774680a4c656e6774680a712c86712d51580a0000005f5f706172656e745f5f712e43080000000000000061712f63706c6f6e652e7365727665722e636f6e74656e740a536974650a713086713151580c000000636f6e7472696275746f72737132680c857133580f0000005f5f616e6e6f746174696f6e735f5f7134430800000000000000d47135682886713651752e
(1 row)

[jimfulton]

See https://github.com/newtdb/db/pull/5

[jimfulton]

The issue was datetime values with time zones.