Tundra: ClassCastException when retrieving object back from Dynamo Datastore using ensure-ks

[johnboy14]

When trying to retrieve the following data structure from the dynamo data store, I keep getting this cast exception

(wcar* (car/hset* "ted" {:name "ted"}))
(wcar* (tundra/dirty dstore "ted"))
(wcar* (car/del "ted"))

(wcar* (tundra/ensure-ks dstore "ted")) = > throws

Exception

clojure.lang.ExceptionInfo: Failed to ensure some key(s)
    data: {"ted" #error {
           :cause "clojure.lang.PersistentArrayMap cannot be cast to [B"
           :via
           [{:type java.lang.ClassCastException
             :message "clojure.lang.PersistentArrayMap cannot be cast to [B"
             :at [taoensso.encore$ba_split invokePrim "encore.clj" 614]}]
           :trace
           [[taoensso.encore$ba_split invokePrim "encore.clj" 614]
            [taoensso.nippy$try_parse_header invoke "nippy.clj" 617]
            [taoensso.nippy$thaw invoke "nippy.clj" 701]
            [taoensso.carmine.tundra.NippyFreezer thaw "tundra.clj" 78]
            [taoensso.carmine.tundra.TundraStore$fn__25108 invoke "tundra.clj" 171]
            [clojure.core$mapv$fn__6727 invoke "core.clj" 6616]
            [clojure.lang.PersistentVector reduce "PersistentVector.java" 333]
            [clojure.core$reduce invoke "core.clj" 6518]
            [clojure.core$mapv invoke "core.clj" 6616]
            [taoensso.carmine.tundra.TundraStore ensure_ks_STAR_ "tundra.clj" 170]
            [taoensso.carmine.tundra$ensure_ks doInvoke "tundra.clj" 62]
            [clojure.lang.RestFn invoke "RestFn.java" 423]
            [com.pav.vote.api.system$eval31000$fn__31001$fn__31002 invoke "form-init8019177790383286520.clj" 1]
            [taoensso.carmine.protocol$with_replies_STAR_ invoke "protocol.clj" 341]
            [com.pav.vote.api.system$eval31000$fn__31001 invoke "form-init8019177790383286520.clj" 1]
            [com.pav.vote.api.system$eval31000 invoke "form-init8019177790383286520.clj" 1]
            [clojure.lang.Compiler eval "Compiler.java" 6782]
            [clojure.lang.Compiler eval "Compiler.java" 6745]
            [clojure.core$eval invoke "core.clj" 3081]
            [clojure.main$repl$read_eval_print__7099$fn__7102 invoke "main.clj" 240]
            [clojure.main$repl$read_eval_print__7099 invoke "main.clj" 240]
            [clojure.main$repl$fn__7108 invoke "main.clj" 258]
            [clojure.main$repl doInvoke "main.clj" 258]
            [clojure.lang.RestFn invoke "RestFn.java" 1523]
            [clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__29718 invoke "interruptible_eval.clj" 48]
            [clojure.lang.AFn applyToHelper "AFn.java" 152]
            [clojure.lang.AFn applyTo "AFn.java" 144]
            [clojure.core$apply invoke "core.clj" 630]
            [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1868]
            [clojure.lang.RestFn invoke "RestFn.java" 425]
            [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 46]
            [clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__29760$fn__29763 invoke "interruptible_eval.clj" 178]
            [clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__29753 invoke "interruptible_eval.clj" 147]
            [clojure.lang.AFn run "AFn.java" 22]
            [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
            [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
            [java.lang.Thread run "Thread.java" 745]]}}

[ptaoussanis]

Hi John,

Could you show me how you're setting up your data store? Also, can I just confirm: by (car/hset* <key> <map>) you mean (car/hset <key> <field> <val>), yes?

[johnboy14]

Sorry about the typo. I am actually using (wcar* (car/hmset* "ted" {:name "ted"})) to persist the data to redis. The Datastore setup is included below.

(def dstore
  (tundra/tundra-store
    (faraday-datastore client-opts
                       {:table :tundratable
                        :key-ns :default})))

(tundra/worker dstore conn-opts
               {:n-threads 3})

[ptaoussanis]

And you've run taoensso.carmine.tundra.faraday/ensure-table to setup the Tundra table?

Could you show me your lein deps :tree output?

[ptaoussanis]

Never mind, this is a bug- investigating now. Thanks for the report!

[johnboy14]

Yes the table seems to be setup ok in Dynamodb.

Heres my dependency tree

lein deps :tree
Possibly confusing dependencies found:
[org.clojure/tools.namespace "0.2.4"]
 overrides
[org.danielsz/system "0.2.0-SNAPSHOT"] -> [ns-tracker "0.3.0"] -> [org.clojure/tools.namespace "0.2.10"]
 and
[midje "1.7.0"] -> [org.clojure/tools.namespace "0.2.7"]

Consider using these exclusions:
[org.danielsz/system "0.2.0-20151120.042720-13" :exclusions [org.clojure/tools.namespace]]
[midje "1.7.0" :exclusions [org.clojure/tools.namespace]]

[liberator "0.13"] -> [hiccup "1.0.3"]
 overrides
[ring/ring-devel "1.3.1"] -> [hiccup "1.0.5"]

Consider using these exclusions:
[ring/ring-devel "1.3.1" :exclusions [hiccup]]

[midje "1.7.0"] -> [clj-time "0.9.0" :exclusions [org.clojure/clojure]]
 overrides
[buddy "0.6.1"] -> [buddy/buddy-auth "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [clj-time "0.10.0"]
 and
[buddy "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [clj-time "0.10.0"]

Consider using these exclusions:
[buddy "0.6.1" :exclusions [clj-time]]
[buddy "0.6.1" :exclusions [clj-time]]

[midje "1.7.0"] -> [slingshot "0.12.1"]
 overrides
[buddy "0.6.1"] -> [buddy/buddy-auth "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [buddy/buddy-core "0.6.0"] -> [slingshot "0.12.2"]
 and
[buddy "0.6.1"] -> [buddy/buddy-hashers "0.6.0"] -> [buddy/buddy-core "0.6.0"] -> [slingshot "0.12.2"]
 and
[buddy "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [buddy/buddy-core "0.6.0"] -> [slingshot "0.12.2"]
 and
[buddy "0.6.1"] -> [buddy/buddy-core "0.6.0"] -> [slingshot "0.12.2"]

Consider using these exclusions:
[buddy "0.6.1" :exclusions [slingshot]]
[buddy "0.6.1" :exclusions [slingshot]]
[buddy "0.6.1" :exclusions [slingshot]]
[buddy "0.6.1" :exclusions [slingshot]]

[compojure "1.4.0"] -> [ring/ring-core "1.4.0"] -> [org.clojure/tools.reader "0.9.1"]
 overrides
[buddy "0.6.1"] -> [buddy/buddy-auth "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [com.taoensso/nippy "2.9.0"] -> [com.taoensso/encore "1.32.0"] -> [org.clojure/tools.reader "0.9.2"]
 and
[buddy "0.6.1"] -> [buddy/buddy-auth "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [com.taoensso/nippy "2.9.0"] -> [org.clojure/tools.reader "0.9.2"]
 and
[buddy "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [com.taoensso/nippy "2.9.0"] -> [com.taoensso/encore "1.32.0"] -> [org.clojure/tools.reader "0.9.2"]
 and
[com.taoensso/faraday "1.8.0"] -> [com.taoensso/nippy "2.9.1"] -> [com.taoensso/encore "1.32.0"] -> [org.clojure/tools.reader "0.9.2"]
 and
[buddy "0.6.1"] -> [buddy/buddy-sign "0.6.1"] -> [com.taoensso/nippy "2.9.0"] -> [org.clojure/tools.reader "0.9.2"]
 and
[com.taoensso/faraday "1.8.0"] -> [com.taoensso/nippy "2.9.1"] -> [org.clojure/tools.reader "0.9.2"]
 and
[com.taoensso/faraday "1.8.0"] -> [com.taoensso/encore "2.18.0"] -> [org.clojure/tools.reader "0.9.2"]

Consider using these exclusions:
[buddy "0.6.1" :exclusions [org.clojure/tools.reader]]
[buddy "0.6.1" :exclusions [org.clojure/tools.reader]]
[buddy "0.6.1" :exclusions [org.clojure/tools.reader]]
[com.taoensso/faraday "1.8.0" :exclusions [org.clojure/tools.reader]]
[buddy "0.6.1" :exclusions [org.clojure/tools.reader]]
[com.taoensso/faraday "1.8.0" :exclusions [org.clojure/tools.reader]]
[com.taoensso/faraday "1.8.0" :exclusions [org.clojure/tools.reader]]

 [buddy "0.6.1"]
   [buddy/buddy-auth "0.6.1"]
     [funcool/cuerdas "0.6.0"]
   [buddy/buddy-core "0.6.0"]
     [org.bouncycastle/bcpkix-jdk15on "1.52"]
     [org.bouncycastle/bcprov-jdk15on "1.52"]
   [buddy/buddy-hashers "0.6.0"]
     [clojurewerkz/scrypt "1.2.0"]
       [com.lambdaworks/scrypt "1.4.0"]
   [buddy/buddy-sign "0.6.1"]
     [funcool/cats "0.6.1"]
 [cheshire "5.5.0"]
   [com.fasterxml.jackson.core/jackson-core "2.5.3"]
   [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.5.3"]
   [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.5.3"]
   [tigris "0.1.1"]
 [clojure-complete "0.2.3" :exclusions [[org.clojure/clojure]]]
 [clojure-msgpack "1.1.2"]
   [org.clojure/test.check "0.7.0"]
 [com.taoensso/carmine "2.12.0" :exclusions [[org.clojure/data.json] [org.clojure/tools.reader]]]
   [com.taoensso/encore "2.19.0"]
   [com.taoensso/nippy "2.10.0"]
     [net.jpountz.lz4/lz4 "1.3"]
     [org.iq80.snappy/snappy "0.4"]
     [org.tukaani/xz "1.5"]
   [com.taoensso/timbre "4.1.4"]
     [io.aviso/pretty "0.1.17"]
   [commons-codec "1.10"]
   [org.apache.commons/commons-pool2 "2.4.2"]
 [com.taoensso/faraday "1.8.0"]
   [com.amazonaws/aws-java-sdk-dynamodb "1.9.39" :exclusions [[joda-time]]]
     [com.amazonaws/aws-java-sdk-core "1.9.39"]
       [com.fasterxml.jackson.core/jackson-databind "2.3.2"]
         [com.fasterxml.jackson.core/jackson-annotations "2.3.0"]
       [commons-logging "1.1.3"]
       [org.apache.httpcomponents/httpclient "4.3.4"]
         [org.apache.httpcomponents/httpcore "4.3.2"]
     [com.amazonaws/aws-java-sdk-s3 "1.9.39"]
       [com.amazonaws/aws-java-sdk-kms "1.9.39"]
   [joda-time "2.8.2"]
 [compojure "1.4.0"]
   [clout "2.1.2"]
     [instaparse "1.4.0" :exclusions [[org.clojure/clojure]]]
   [medley "0.6.0"]
   [org.clojure/tools.macro "0.1.5"]
   [ring/ring-codec "1.0.0"]
   [ring/ring-core "1.4.0"]
     [commons-fileupload "1.3.1"]
     [commons-io "2.4"]
     [crypto-equality "1.0.0"]
     [crypto-random "1.2.0"]
     [org.clojure/tools.reader "0.9.1"]
 [environ "1.0.0"]
 [http-kit "2.1.19"]
 [liberator "0.13"]
   [hiccup "1.0.3"]
   [org.clojure/data.csv "0.1.2"]
   [org.clojure/data.json "0.2.1"]
 [midje "1.7.0" :scope "test"]
   [clj-time "0.9.0" :exclusions [[org.clojure/clojure]]]
   [colorize "0.1.1" :scope "test" :exclusions [[org.clojure/clojure]]]
   [dynapath "0.2.0" :scope "test"]
   [flare "0.2.8" :scope "test" :exclusions [[org.clojure/clojure]]]
     [org.clojars.brenton/google-diff-match-patch "0.1" :scope "test"]
   [ordered "1.2.0" :scope "test" :exclusions [[org.clojure/clojure]]]
   [org.clojure/core.unify "0.5.2" :scope "test" :exclusions [[org.clojure/clojure]]]
   [org.clojure/math.combinatorics "0.1.1" :scope "test"]
   [org.tcrawley/dynapath "0.2.3" :scope "test"]
   [slingshot "0.12.1"]
   [swiss-arrows "1.0.0" :scope "test" :exclusions [[org.clojure/clojure]]]
 [org.clojure/clojure "1.7.0"]
 [org.clojure/core.async "0.1.346.0-17112a-alpha"]
   [org.clojure/tools.analyzer.jvm "0.1.0-beta12"]
     [org.clojure/core.memoize "0.5.6"]
       [org.clojure/core.cache "0.6.3"]
         [org.clojure/data.priority-map "0.0.2"]
     [org.clojure/tools.analyzer "0.1.0-beta12"]
     [org.ow2.asm/asm-all "4.1"]
 [org.clojure/tools.logging "0.3.1"]
 [org.clojure/tools.namespace "0.2.4" :scope "test"]
 [org.clojure/tools.nrepl "0.2.5"]
 [org.danielsz/system "0.2.0-20151120.042720-13"]
   [com.stuartsierra/component "0.2.3"]
     [com.stuartsierra/dependency "0.1.1"]
   [ns-tracker "0.3.0"]
     [org.clojure/java.classpath "0.2.2"]
   [reloaded.repl "0.1.0"]
 [prismatic/schema "0.4.3"]
 [ring-cors "0.1.7"]
 [ring-mock "0.1.5" :scope "test"]
 [ring-server "0.3.1"]
   [org.clojure/core.incubator "0.1.0"]
   [ring-refresh "0.1.2"]
     [watchtower "0.1.1"]
   [ring "1.2.1"]
     [ring/ring-jetty-adapter "1.2.1"]
       [org.eclipse.jetty/jetty-server "7.6.8.v20121106"]
         [org.eclipse.jetty.orbit/javax.servlet "2.5.0.v201103041518"]
         [org.eclipse.jetty/jetty-continuation "7.6.8.v20121106"]
         [org.eclipse.jetty/jetty-http "7.6.8.v20121106"]
           [org.eclipse.jetty/jetty-io "7.6.8.v20121106"]
             [org.eclipse.jetty/jetty-util "7.6.8.v20121106"]
     [ring/ring-servlet "1.2.1"]
 [ring/ring-devel "1.3.1" :scope "test"]
   [clj-stacktrace "0.2.7" :scope "test"]
 [ring/ring-json "0.3.1"]

[ptaoussanis]

Okay, just pushed [com.taoensso/carmine "2.12.1"] which should resolve this - could you possibly confirm?

[johnboy14]

Yeah it works now. Thanks for the quick turnaround. Just curious is bytes the only form of serialization available to the dynamo data store?. I was hoping I could persist it to dynamo as a json string to ease the pain of eye balling a dynamo table for debugging purposes.

[ptaoussanis]

No problem, appreciate the report.

And yeah- bytes is the only form of serialization, sorry. Carmine uses Nippy for serialization since it's fast and flexible. Visual debugging's one thing you give up in exchange, though you could pass any serialized vals to Nippy for manual thawing if you like.

[johnboy14]

http://ptaoussanis.github.io/carmine/taoensso.carmine.tundra.html#var-tundra-store

I see an option to provide our own freezer here. The problem we are having is we are using msgpack to serialize/deserialize other data and we are trying to avoid a mix match of serialization/deserialization techniques. I assume this is strictly a nippy freezer we can provide.?

[ptaoussanis]

I assume this is strictly a nippy freezer?

No no, that is a fully functional extension point if you want to avoid Nippy altogether - but you'd need an alternative de/serialization tool that handles all the necessary Clojure data types that you expect.

Nippy's very well suited here though, and very well tested- I'd be cautious of swapping it out unless you have a strong reason to (and only if you have a strong alternative to swap in).

You can see the IFreezer protocol for details.