search

taoensso / nippy

The fastest serialization library for Clojure
https://www.taoensso.com/nippy
Eclipse Public License 1.0
1.04k stars 60 forks source link

Error while running thaw #14

Closed adityo closed 11 years ago

adityo commented 11 years ago

Hey,

I get the following error when running the sample call (nippy/freeze nippy/stress-data) in my repl. When i thaw the above result, i get the error given below.

CorruptionException Invalid copy offset for opcode starting at 23 org.iq80.snappy.SnappyDecompressor.decompressAllTags (SnappyDecompressor.java:165)

running clojure 1.5.1 nippy 2.0.0-RC1

ptaoussanis commented 11 years ago

Hi adityo,

Could you get me some info on your environment, and the exact code you're trying to execute?

Thanks!

adityo commented 11 years ago

Hey Peter,

im currently running emacs 24, with clojure 1.5.1 , nrepl instead of slime/swank on mac osx lion.

I have been using carmine to store data into redis. On carmine "1.12.0", i was serializing a sequence of clojure maps into json and then writing to redis. each clojure map had a key whose value was a joda time date obj.

with carmine 2.0.0-beta5, nippy is also included and i thought i might give it a try (hopefully i wouldn't have to serialize my objects to json). since latest carmine fetches latest nippy i simply included it in my project. i ran the following commands on my repl.

(def frozen-stress-data (nippy/freeze nippy/stress-data)) and (nippy/thaw frozen-stress-data)

which led to the following error CorruptionException Invalid copy offset for opcode starting at 23 org.iq80.snappy.SnappyDecompressor.decompressAllTags (SnappyDecompressor.java:165)

that is the most basic example provided on the github page.

i also tried it on my data set, the freeze command works however the thaw command causes the above error.

--regards Adie

ptaoussanis commented 11 years ago

Well thanks a lot for reporting this Adie!

Not quite sure yet what the problem is since I'm struggling to reproduce it from my side.

i also tried it on my data set, the freeze command works however the thaw command causes the above error.

So you're saying you get this error for anything you freeze and try to thaw again?

Could you tell me what happens with each of the following?:

Also could you tell me what OS you're on and what version of Java you're running?

Thanks a lot for your help - really want to get to the bottom of this.

Cheers! :-)

adityo commented 11 years ago

Hey Peter,

I am on Mac OSX Lion and Java version is _Java(TM) SE Runtime Environment (build 1.6.051-b11-456-11M4508) Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-456, mixed mode)

Here are the outputs for each

Additionally

Here is a sample of what im trying to freeze and thaw with the outputs i'm getting.

or

--regards Adityo

ptaoussanis commented 11 years ago

(nippy/thaw (nippy/freeze nippy/stress-data {:compressor nil})) => CorruptionException ...

This one is leading me to believe the issue we're seeing may be a result of some difference in the way your JVM is processing byte arrays for some reason. Almost certainly a bug on my end, but I still haven't been able to reproduce it.

I've just pushed a debug version: [com.taoensso.debug/nippy "2.0.0-beta3"] (note the changed group-id).

When you get a chance, would you mind rerunning the same tests again with this version? It'll produce some println output which you can copy me and should help diagnose the cause.

Again, thanks so much for your help with this!

ptaoussanis commented 11 years ago

Yeah, sorry - I'm out of ideas w/o further info. Have tried a number of JVMs and a number of datatypes w/o being able to reproduce this. Have gone over all the relevant code, and nothing's sticking out as obviously broken.

These two in particular surprise me:

The only way the first one can be happening is if the header is somehow corrupt (otherwise thaw wouldn't even be attempting decompression). In that case though you'd be getting an "Uncompressed data?" exception.

The second one, likewise, is confounding. There's nothing in the stress data that requires read-eval. Even if the header were corrupt (therefore corrupting the data), I don't see why this wouldn't raise the same "Uncompressed data?" exception.

Have nothing else I can think of to try on my end, so I'll wait on your response to the debug version's output.

Cheers!

adityo commented 11 years ago

Hey Peter,

Somehow i wasn't getting any output when using nrepl so switched to slime-repl.

Here are the following outputs.

i shall try and run the same on my colleague's machine as soon as i get in to work. Hope i have not messed up something on my system only.

--regards Adityo

ptaoussanis commented 11 years ago

Hi Adityo, okay, that was very helpful - think we're getting somewhere!

My hunch is that you've got some kind of custom data-reader in your environment that is fiddling with the nippy/stress-data in an unexpected way. That would potentially explain every error we've seen.

Here's a (hopefully final) set of tests to run:

If this one works, it confirms the diagnosis. No need to try the rest.

This one should throw an exception describing (hopefully) the actual cause, assuming my hunch above is incorrect.

If the diagnosis was confirmed: Basically it looks like something is interfering with the way your Reader is interpreting dates. What happens if you evaluate (java.util.Date.) at the REPL? What about (read-string (pr-str (java.util.Date.)))? What is the value of your *data-readers*?

(read-string (pr-str (java.util.Date.))) should work (regardless of any custom data readers). If that's not working, then that's what you need to debug.

If the diagnosis was NOT confirmed: Send me the output from the above tests and let's go from there.

Cheers :-)

adityo commented 11 years ago

Hey Peter,

Diagnosis confirmed. Something is interfering with the way my Reader is interpreting dates.

To confirm i ran a very basic project fetching only nippy as a dependency and the example (nippy/thaw (nippy/freeze nippy/stress-data)) worked perfectly.

So some library i am fetching for my project seems to be adding in a custom reader, and we think we might have the found the problem (thanks to the efforts of my colleague Kiran).

Monger is the culprit. https://github.com/michaelklishin/monger/blob/master/src/clojure/monger/joda_time.clj#L42

From Line 42 there are two custom print-dup, one for java.util.Date and the other for joda time object and i guess these are messing up.

I would suggest you fetch monger in your test project to confirm this behavior.

--regards Adityo

ptaoussanis commented 11 years ago

Yup, that could very well be the cause (assuming you've got no other libs doing the same thing). To confirm, try: (nippy/thaw (nippy/freeze nippy/stress-data) {:read-eval true})) (it should work).

The problem is that this method is using a "#=" (reader eval) call which is normally disabled by default in Nippy for security. I'll mention this to Michael (Monger author).

adityo commented 11 years ago

Yes confirmed,

(nippy/thaw (nippy/freeze nippy/stress-data) {:read-eval? true}) works.

Thanks a lot for your patience.

--regards Adityo

ptaoussanis commented 11 years ago

Okay, great. I would strongly recommend against using that workaround btw. Better to get the underlying problem solved. Enabling *read-eval* can expose you to serious security issues since it can allow an attacker to basically evaluate any code in your JVM instance.

Thanks a lot for your patience.

Absolutely no problem! Happy I could help :-)