Question: "right" way to thaw an input-stream

[martinklepsch]

Hey! Let's say I have an input stream like this:

(:input-stream (s3/get-object bucket "some-thing"))

What would be the right way to thaw it? I tried converting the input-stream to a DataInputStream (and using thaw-from-in! but that resulted in failed deserializtion:

ExceptionInfo Unrecognized type id (78). Data frozen with newer Nippy version?  

I ended up doing something like below involving commons' IOUtils class:

(-> (s3/get-object bucket "some-thing")
      :input-stream
      IOUtils/toByteArray
      nippy/thaw)

Maybe this would be something useful for the readme or this issue will be helpful to others.

[DerGuteMoritz]

Here's a concise repro case:

(-> (nippy/freeze [:hello 123])
    (java.io.ByteArrayInputStream.)
    (java.io.DataInputStream.)
    (nippy/thaw-from-in!))

[DerGuteMoritz]

Going through freeze-to-out! works, though:

(let [out (java.io.ByteArrayOutputStream.)]
  (nippy/freeze-to-out! (java.io.DataOutputStream. out) [:hello 123])
  (-> (.toByteArray out)
      (java.io.ByteArrayInputStream.)
      (java.io.DataInputStream.)
      (nippy/thaw-from-in!)))
;; => [:hello 123]

[ptaoussanis]

Hey Martin, Moritz-

So as the docstrings mention, thaw-from-in! and freeze-to-out! are low-level utils.

Specifically:

• freeze-to-out! freezes without any compression, encryption, or addition of the Nippy header.
• thaw-from-in! thaws assuming any decryption, decompression, or processing of the Nippy header.

So you can't mix freeze-to-out! with thaw or thaw-from-in! with freeze.

Going through freeze-to-out! works, though

So long as you consistently use the low/high level utils for freezing + thawing you'll be fine.

Does that help / make sense?

[martinklepsch]

Ah that's really good to know, I was absolutely unaware that thaw/freeze and their low level counterparts are not directly compatible. 😅 Thanks for clearing that up :)

So with regards to the original question this means that it should have worked if I

• use freeze-from-out!
• and then turning the input steam into a DataInputStream before passing it to thaw-from-in!

Correct?

[ptaoussanis]

I was absolutely unaware that thaw/freeze and their low level counterparts are not directly compatible

It's an intrinsic incompatibility; the regular thaw/freeze support encryption + compression, but those require[1] the whole byte-array to be available up-front.

Correct?

Sounds good 👍

[1] Technically we could try do streaming encryption+compression, but that has its own tradeoffs - including limiting the encryption+compression options.

[martinklepsch]

Thanks for all the explanations :)