No reader provided for custom type ID: 98

[dsbw]

Much like problem #75, except I'm not using a custom storage type.

My code (well, the code I've inherited, natch) calls freeze and thaw, once each exactly, with no other nippy interaction. I'm trying to pull that out from a different Clojure app and getting the above error. At first I thought it was a version problem, since the original, working code uses 2.5.2, but I switched back to 2.5.2 (confirmed with "lein deps :tree") and still have the issue.

The code I'm trying to run is:

(def rec (mc/find-one-as-map db "coll" {:time {ops/$gt (java.util.Date. 1454284800000)}})) (println "***" (:bin rec)) (println (nippy/thaw (:bin rec)))

Where bin was originally frozen with Nippy. The full dump when run at the command line is:

***#object[[B 0x3b979721 [B@3b979721] Exception in thread "main" java.lang.Exception: Thaw failed: Encrypted data or w rong compressor?, compiling:(mon2sql/mongo.clj:120:1) at clojure.lang.Compiler.load(Compiler.java:7239) at clojure.lang.RT.loadResourceScript(RT.java:371) at clojure.lang.RT.loadResourceScript(RT.java:362) at clojure.lang.RT.load(RT.java:446) at clojure.lang.RT.load(RT.java:412) at clojure.core$load$fn5448.invoke(core.clj:5866) at clojure.core$load.doInvoke(core.clj:5865) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invoke(core.clj:5671) at clojure.core$load_lib$fn5397.invoke(core.clj:5711) at clojure.core$load_lib.doInvoke(core.clj:5710) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invoke(core.clj:632) at clojure.core$load_libs.doInvoke(core.clj:5749) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invoke(core.clj:632) at clojure.core$require.doInvoke(core.clj:5832) at clojure.lang.RestFn.invoke(RestFn.java:408) at mon2sql.core$eval504.invoke(core.clj:5) at clojure.lang.Compiler.eval(Compiler.java:6782) at clojure.lang.Compiler.load(Compiler.java:7227) at clojure.lang.RT.loadResourceScript(RT.java:371) at clojure.lang.RT.loadResourceScript(RT.java:362) at clojure.lang.RT.load(RT.java:446) at clojure.lang.RT.load(RT.java:412) at clojure.core$load$fn5448.invoke(core.clj:5866) at clojure.core$load.doInvoke(core.clj:5865) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invoke(core.clj:5671) at clojure.core$load_lib$fn5397.invoke(core.clj:5711) at clojure.core$load_lib.doInvoke(core.clj:5710) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invoke(core.clj:632) at clojure.core$load_libs.doInvoke(core.clj:5749) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invoke(core.clj:632) at clojure.core$require.doInvoke(core.clj:5832) at clojure.lang.RestFn.invoke(RestFn.java:408) at user$eval5$fn__7.invoke(form-init2544198019161173763.clj:1) at user$eval5.invoke(form-init2544198019161173763.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6782) at clojure.lang.Compiler.eval(Compiler.java:6772) at clojure.lang.Compiler.load(Compiler.java:7227) at clojure.lang.Compiler.loadFile(Compiler.java:7165) at clojure.main$load_script.invoke(main.clj:275) at clojure.main$init_opt.invoke(main.clj:280) at clojure.main$initialize.invoke(main.clj:308) at clojure.main$null_opt.invoke(main.clj:343) at clojure.main$main.doInvoke(main.clj:421) at clojure.lang.RestFn.invoke(RestFn.java:421) at clojure.lang.Var.invoke(Var.java:383) at clojure.lang.AFn.applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var.java:700) at clojure.main.main(main.java:37) Caused by: java.lang.Exception: Thaw failed: Encrypted data or wrong compressor?

    at taoensso.nippy$thaw$ex__2319.doInvoke(nippy.clj:392)
    at clojure.lang.RestFn.invoke(RestFn.java:423)
    at taoensso.nippy$thaw$try_thaw_data__2323.invoke(nippy.clj:410)
    at taoensso.nippy$thaw.doInvoke(nippy.clj:426)
    at clojure.lang.RestFn.invoke(RestFn.java:410)
    at mon2sql.mongo$eval2474.invoke(mongo.clj:120)
    at clojure.lang.Compiler.eval(Compiler.java:6782)
    at clojure.lang.Compiler.load(Compiler.java:7227)
    ... 53 more

Caused by: java.lang.Exception: Thaw failed against type-id: 27 at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:368) at taoensso.nippy$thaw_from_streamBANG.doInvoke(nippy.clj:374) at clojure.lang.RestFn.invoke(RestFn.java:410) at taoensso.nippy$thaw$try_thaw_data2323.invoke(nippy.clj:405) ... 58 more Caused by: java.lang.Exception: Thaw failed against type-id: 27 at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:368) at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:321) ... 61 more Caused by: java.lang.Exception: Thaw failed against type-id: 24 at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:368) at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:321) ... 62 more Caused by: java.lang.Exception: Thaw failed against type-id: -98 at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:368) at taoensso.nippy$thaw_from_stream$fn2219.invoke(nippy.clj:322) at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:322) ... 63 more Caused by: java.lang.Exception: No reader provided for custom type ID: 98 at taoensso.nippy$thaw_from_stream.invoke(nippy.clj:364) ... 65 more

[ptaoussanis]

Hi Blake,

No reader provided for custom type ID: 98

This error implies that something was frozen with a custom id (98), but that the relevant thaw routine hasn't been configured with extend-thaw, etc.

If you're 100% convinced that no custom ids were ever written, then unfortunately the only other explanation I can think of is data corruption somehow. Have you confirmed that you're using the correct compression+encryption settings?

(def rec (mc/find-one-as-map db "coll" {:time {ops/$gt (java.util.Date. 1454284800000)}}))
(println "***" (:bin rec))
(println (nippy/thaw (:bin rec)))

This unfortunately doesn't mean much to me since I'm not sure what mc/find-one-as-map is or what it's returning.

Could you show me the output from calling inspect-ba on the frozen byte array that's throwing this error?

[ptaoussanis]

Update:

since the original, working code uses 2.5.2, but I switched back to 2.5.2 (confirmed with "lein deps :tree") and still have the issue.

I'd suggesting using the newest version of Nippy available. All newer versions are backwards compatible but may offer better debug info, etc.

[dsbw]

Whoops! Sorry, that "mc" is from Monger. The frozen data is stored as a binData in Mongo.

I don't think corruption is the issue, because I'm pretty sure the program that freezes this data also thaws it, though I will definitely double-check that and try to see if there's a custom type I'm not seeing.

As for inspect-ba, well that's interesting:

{:known-wrapper nil, :nippy2-header {:version 1, :compressed? true, :encrypted? false}, :thawable? false, :unwrapped-ba #object["[B" 0x4552d6d9 "[B@4552d6d9"], :data-ba #object["[B" 0x4971d710 "[B@4971d710"], :unwrapped-size 71064, :ba-size 71064, :data-size 71060}

Why would thawable? be false?

[dsbw]

The "thawable?" function just tries to unthaw and catch the error, so of course it would return false...

[ptaoussanis]

If you're certain there's no custom types being written (please grep your codebase for extend-freeze and Freezable) then next step would be trying to get me a reproducible example that includes the data that you're freezing that throws when you try thaw it.

(def rec (mc/find-one-as-map db "coll" {:time {ops/$gt (java.util.Date. 1454284800000)}}))
(println "***" (:bin rec))
(println (nippy/thaw (:bin rec)))

I'd start by trying to rule out that any of the code/libs you're using are modifying Nippy's byte array in any way. You might want to compare the byte sequence before and after transfer to the db.

[dsbw]

The situation is: a) the data is frozen in Mongo; b) it's retrievable in mongo from the code that freezes it; c) it's not retrievable elsewhere; d) No variations, extensions, or anything more than simple freeze/thaw are in play.

So, I'm going to go with this being a Mongo problem. Thanks. If I can come up with a genuine Nippy issue, I'll be back.

[ptaoussanis]

Hi Blake,

The situation is: a) the data is frozen in Mongo; b) it's retrievable in mongo from the code that freezes it;

Not sure I exactly follow your meaning here. In any case, yes- one of the first things you might like to check is that Mongo isn't modifying Nippy's byte array in any way (e.g. changing the encoding, adding a prefix, etc.). If it does modify Nippy's byte array, then you could very well see errors like No reader provided for custom type ID: 98 since Nippy's byte array will be corrupt.

I'd start by trying to write a known arbitrary byte array to Mongo, reading it back and comparing the result.

Best of luck!

[dsbw]

I have discovered the following: The code that stores the data was written using Clojure 1.5.1. The code trying to retrieve it was using 1.7.0. If I use 1.5.1 in the retrieving code, it works. I got the idea when a co-worker reminded me we had explicitly defined big-decimals, which weren't part of the language until Clojure 1.6.0.

These weren't given any special treatment, i.e., we didn't write an extend-freeze or Freezable, but the 2.5.2 version of Nippy depended on org.clojure/tools.reader 0.7.10, whereas 2.11.1 does not. I believe this is the root of the problem.

Not sure how interested you are in pursuing this but thought I'd put my results up for any others faced with a similar issue.

[ptaoussanis]

Hi Blake, thanks for the follow up.

we had explicitly defined big-decimals

Could you clarify here what you mean exactly? Upgrading Clojure versions shouldn't inherently cause any problems with Nippy.

The only way you should be able to run into problems is if you've defined a custom type that was serialized in environment A, then a deserialization attempt is made in environment B where the same type isn't available.

So, for example: if you use extend-freeze or a custom Clojure reader type in environment A without the corresponding extend-thaw or custom Clojure reader type in environment B.

If you make sure that environment B has all the necessary types available (including extend-thaws and custom Clojure reader types), then environment B should be able to thaw anything from environment A if environment B has >= A's versions of Nippy and Clojure.

Does that help / make sense?

[dsbw]

Well, let me be explicit as to what I've done:

-> Some Clojure 1.5.1 code I've inherited as part of a rather large project froze data with Nippy 2.5.2. It saved the frozen data using Monger 1.7.0.

-> I created a new simple project using Clojure 1.7.0 that tried to thaw the data with Nippy 2.11.0 using Monger 3.0.1. That's when the problem occurred.

-> I rolled the Nippy and the Monger back to match the original project, with no success. Still couldn't unthaw.

-> I rolled the Clojure back to 1.5.1 and, voila, all the data.

-> I rolled the Nippy and Monger forward again to 2.11.0 and 3.0.1, respectively, and I can still get to the data.

-> I upgrade Clojure back to 1.7.0 and get "Thaw failed: Decryption/decompression failure, or data unfrozen/damaged."

-> I upgrade Clojure to 1.8.0 for giggles, same result. Same with 1.6.0.

My conclusion from this is that the problem is that something changed between Clojure 1.5.1 and Clojure 1.6.0, that affects Nippy. Looking at the old code, I see a dependency in Nippy 2.5.2 for [clojure.tools.reader.edn :as edn] which does not appear in 2.11.0. I surmise the problem comes from that.

There is no extend-freeze/extend-thaw. There might be a custom reader type, which I'll investigate next. However, whatever the conflict is, it doesn't exist in 1.5.1. Since I can now thaw, I may be able to get you a sample that you can freeze/thaw in 1.5.1, but not thaw in 1.6+.

[ptaoussanis]

Hi Blake,

My conclusion from this is that the problem is that something changed between Clojure 1.5.1 and Clojure 1.6.0, that affects Nippy. Looking at the old code, I see a dependency in Nippy 2.5.2 for [clojure.tools.reader.edn :as edn] which does not appear in 2.11.0. I surmise the problem comes from that.

As I explained, this should not be a problem unless types have been configured in one of the two ways I described.

I got the idea when a co-worker reminded me we had explicitly defined big-decimals, which weren't part of the language until Clojure 1.6.0.

This is the salient bit that requires investigating re: what you/your co-worker means by "we had explicitly defined big-decimals".

To recap my earlier suggestions if you'd like to proceed:

1. Investigate "we had explicitly defined big-decimals". If custom reader types were frozen with the previous version of your project, they need to be available in all thaw environments.
2. I'd confirm that bytes written to and read from your database don't undergo any changes. (In particular: between your old and new environments). Just compare a random byte sequence before/after storage.
3. Confirm that no compression or encryption settings have changed between environments.
4. I'd suggest forwarding me an example of a frozen byte sequence that can't be unthawed. I can look at the sequence to identify what types are in there, and to rule out other potential problems.

Otherwise, will unfortunately need to excuse myself from this exchange for the moment since I have my hands full with some urgent work. Happy you've found a way to access the data at least. Best of luck with the continued debugging!

Cheers :-)