thaw-from-in! throws an error while thaw works

[schmir]

I'm trying to deserialize an object from a file. When I read the object from disk, and call thaw on it, I can deserialise it without problems. But when calling thaw-from-in I get an error "No reader provided for custom type with internal id: 78"

Here's a short REPL session:

steinmetz.qlauf.ueberweisung> (time (keys (nippy/thaw (slurp-bytes "frozen-state"))))
"Elapsed time: 1817.759265 msecs"
;; => (:qparams)
steinmetz.qlauf.ueberweisung> (time (keys (nippy/thaw-from-in! (java.io.DataInputStream. (io/input-stream "frozen-state")))))
clojure.lang.ExceptionInfo: Thaw failed against type-id: 78
clojure.lang.ExceptionInfo: No reader provided for custom type with internal id: 78

where slurp-bytes is

(defn slurp-bytes
  "Slurp the bytes from a slurpable thing"
  [x]
  (with-open [out (java.io.ByteArrayOutputStream.)]
    (clojure.java.io/copy (clojure.java.io/input-stream x) out)
    (.toByteArray out)))

Am I doing something wrong or is that a bug?

[ptaoussanis]

Hi Ralf, thaw-from-in! is a low level util that you'll normally want to avoid unless you understand exactly what it's doing (in this case tripping up because it's assuming the Nippy header has already been consumed when it hasn't).

Is there a reason you're avoiding the standard thaw API?

[schmir]

I assumed thaw-from-in! was a way to thaw from a file. The only reason I had for trying thaw-from-in! was that I had to lookup the above slurp-bytes function and assumed that nippy already had a way to thaw from a file.

Anyway, sorry for the noise and thanks for nippy (and timbre - while I'm here)!

[ptaoussanis]

Anyway, sorry for the noise and thanks for nippy (and timbre - while I'm here)!

No problem, and you're very welcome!

Yeah, no - the easiest way to go is usually with the thaw function which takes a byte array.

The thaw-from-in! function is useful if you're doing streaming de/serialization, but it forces some limitations on your frozen data (for example: you'd need to disable the Nippy header and any default compression).

I've updated the docstrings to make this a little clearer, so thanks for bringing it to my attention.

[ptaoussanis]

Oh, just as an aside: if you're calling time there to try benchmark the thaw call - you might get inconsistent results with JVM warmup, etc.

Instead, try (taoensso.encore/quick-bench 10000 (keys (nippy/thaw (slurp-bytes "frozen-state"))))

You'll already have encore as a dependency via Nippy. The quick-bench macro just runs a bunch of timed laps to cut down on measurement noise.