taoensso / carmine

Redis client + message queue for Clojure
https://www.taoensso.com/carmine
Eclipse Public License 1.0
1.15k stars 130 forks source link

Problems using carmine store #288

Closed Pederaugust closed 8 months ago

Pederaugust commented 8 months ago

Hey, love your contributions to open source. I have setup my ring-server to use the carmine-store. Context: I am running redis in docker. For development I use this configuration: {:spec {:uri "redis://0.0.0.0:6379/0"}}

I am getting this error:

                                org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run        QueuedThreadPool.java: 1034
                                    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob        QueuedThreadPool.java:  883
                  org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run  ReservedThreadExecutor.java:  409
                                org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run          EatWhatYouKill.java:  131
                         org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce          EatWhatYouKill.java:  173
                          org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce          EatWhatYouKill.java:  315
                            org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask          EatWhatYouKill.java:  338
                                               org.eclipse.jetty.io.ChannelEndPoint$1.run         ChannelEndPoint.java:  104
                                               org.eclipse.jetty.io.FillInterest.fillable            FillInterest.java:  105
                           org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded      AbstractConnection.java:  311
                                       org.eclipse.jetty.server.HttpConnection.onFillable          HttpConnection.java:  277
                                              org.eclipse.jetty.server.HttpChannel.handle             HttpChannel.java:  479
                                            org.eclipse.jetty.server.HttpChannel.dispatch             HttpChannel.java:  732
                                     org.eclipse.jetty.server.HttpChannel.lambda$handle$1             HttpChannel.java:  487
                                                   org.eclipse.jetty.server.Server.handle                  Server.java:  516
                                   org.eclipse.jetty.server.handler.HandlerWrapper.handle          HandlerWrapper.java:  127
ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle
                                                      ring.adapter.jetty/proxy-handler/fn                    jetty.clj:   27
                                                    ring.middleware.params/wrap-params/fn                   params.clj:   75
                                                                 compojure.core/routes/fn                     core.clj:  192
                                                                       clojure.core/apply                     core.clj:  669
                                                                                      ...
                                                                   compojure.core/routing                     core.clj:  182
                                                                   compojure.core/routing                     core.clj:  185
                                                                        clojure.core/some                     core.clj: 2718
                                                                compojure.core/routing/fn                     core.clj:  185
                                              ring.middleware.refresh/wrap-with-script/fn                  refresh.clj:   81
                                                    ring.middleware.reload/wrap-reload/fn                   reload.clj:   39
                                                                                      ...
                                                        ring.middleware.cors/wrap-cors/fn                    cors.cljc:  205
                                                         ring.middleware.cors/handle-cors                    cors.cljc:  177
                                        ring.middleware.not-modified/wrap-not-modified/fn             not_modified.clj:   61
                                  ring.middleware.default-charset/wrap-default-charset/fn          default_charset.clj:   31
                                        ring.middleware.content-type/wrap-content-type/fn             content_type.clj:   34
                                                    ring.middleware.params/wrap-params/fn                   params.clj:   75
                                    ring.middleware.keyword-params/wrap-keyword-params/fn           keyword_params.clj:   53
                                                                 compojure.core/routes/fn                     core.clj:  192
                                                                       clojure.core/apply                     core.clj:  669
                                                                                      ...
                                                                   compojure.core/routing                     core.clj:  182
                                                                   compojure.core/routing                     core.clj:  185
                                                                        clojure.core/some                     core.clj: 2718
                                                                compojure.core/routing/fn                     core.clj:  185
                               ring.middleware.resource/wrap-resource-prefer-resources/fn                 resource.clj:   25
                                                  ring.middleware.cookies/wrap-cookies/fn                  cookies.clj:  192
                                ring.middleware.multipart-params/wrap-multipart-params/fn         multipart_params.clj:  188
                               ring.middleware.multipart-params/handle-request-and-errors         multipart_params.clj:  134
                         ring.middleware.multipart-params/handle-request-and-errors/fn/fn         multipart_params.clj:  136
                                      ring.middleware.nested-params/wrap-nested-params/fn            nested_params.clj:   89
                                    ring.middleware.keyword-params/wrap-keyword-params/fn           keyword_params.clj:   53
                                                  ring.middleware.session/wrap-session/fn                  session.clj:  113
                                                 ring.middleware.session/session-response                  session.clj:   75
                                            ring.middleware.session/bare-session-response                  session.clj:   53
                                  taoensso.carmine.ring.CarmineSessionStore/write-session                     ring.clj:   20
                                             taoensso.carmine.ring.CarmineSessionStore/fn                     ring.clj:   20
                                          taoensso.carmine.ring.CarmineSessionStore/fn/fn                     ring.clj:   20
                                                  taoensso.carmine.protocol/-with-replies                 protocol.clj:  349
                                       taoensso.carmine.ring.CarmineSessionStore/fn/fn/fn                     ring.clj:   22
                                                                   taoensso.carmine/setex                  carmine.clj:  264
                                                taoensso.carmine.commands/enqueue-request                 commands.clj:  237
                                                                        clojure.core/mapv                     core.clj: 6970
                                                                      clojure.core/reduce                     core.clj: 6885
                                                                                      ...
                                                                     clojure.core/mapv/fn                     core.clj: 6979
                                                 taoensso.carmine.protocol/eval14205/fn/G                 protocol.clj:   63
                                                   taoensso.carmine.protocol/eval14226/fn                 protocol.clj:   76
                                                    taoensso.nippy.tools/eval14127/freeze                    tools.clj:   50
                                                    taoensso.nippy.tools/eval14127/freeze                    tools.clj:   58
                                                                    taoensso.nippy/freeze                    nippy.clj: 1343
                                                        taoensso.nippy/call-with-bindings                    nippy.clj: 1284
                                                                 taoensso.nippy/freeze/fn                    nippy.clj: 1361
                                                              taoensso.nippy/freeze/fn/fn                    nippy.clj: 1361
                                                            taoensso.nippy/eval13521/fn/G                    nippy.clj:  718
                                                              taoensso.nippy/eval13540/fn                    nippy.clj:  726
                                                            taoensso.nippy/eval13493/fn/G                    nippy.clj:  717
                                                              taoensso.nippy/eval13795/fn                    nippy.clj: 1188
                                                                 taoensso.nippy/write-map                    nippy.clj:  926
                                                                   clojure.core/reduce-kv                     core.clj: 6919
                                                              clojure.core.protocols/fn/G                protocols.clj:  175
                                                                          clojure.core/fn                     core.clj: 6908
                                                                                      ...
                                                              taoensso.nippy/write-map/fn                    nippy.clj:  926
                                                           taoensso.nippy/write-map/fn/fn                    nippy.clj:  929
                                                            taoensso.nippy/eval13521/fn/G                    nippy.clj:  718
                                                              taoensso.nippy/eval13540/fn                    nippy.clj:  725
                                                            taoensso.nippy/eval13493/fn/G                    nippy.clj:  717
                                                              taoensso.nippy/eval13795/fn                    nippy.clj: 1188
                                                                 taoensso.nippy/write-map                    nippy.clj:  926
                                                                   clojure.core/reduce-kv                     core.clj: 6919
                                                              clojure.core.protocols/fn/G                protocols.clj:  175
                                                                          clojure.core/fn                     core.clj: 6908
                                                                                      ...
                                                              taoensso.nippy/write-map/fn                    nippy.clj:  926
                                                           taoensso.nippy/write-map/fn/fn                    nippy.clj:  929
                                                            taoensso.nippy/eval13521/fn/G                    nippy.clj:  718
                                                              taoensso.nippy/eval13540/fn                    nippy.clj:  726
                                                            taoensso.nippy/eval13493/fn/G                    nippy.clj:  717
                                                              taoensso.nippy/eval13835/fn                    nippy.clj: 1260
                                                         taoensso.nippy/throw-unfreezable                    nippy.clj: 1020
clojure.lang.ExceptionInfo: Unfreezable type: class next.jdbc.result_set$navize_row$fn__17909
    as-str: "#object[next.jdbc.result_set$navize_row$fn__17909 0x621cc708 \"next.jdbc.result_set$navize_row$fn__17909@621cc708\"]"
      type: next.jdbc.result_set$navize_row$fn__17909

I am not sure why next gives an error when I use the carmine store? When I use the default store everything works fine.

ptaoussanis commented 8 months ago

@Pederaugust Hi there! Thanks for the kind words :-)

So the key info in that stack trace is here:

taoensso.nippy/throw-unfreezable                    nippy.clj: 1020
clojure.lang.ExceptionInfo: Unfreezable type: class next.jdbc.result_set$navize_row$fn__17909
    as-str: "#object[next.jdbc.result_set$navize_row$fn__17909 0x621cc708 \"next.jdbc.result_set$navize_row$fn__17909@621cc708\"]"
      type: next.jdbc.result_set$navize_row$fn__17909

Basically the problem is that you're trying to put something into your Ring session (a next.jdbc type) that Carmine doesn't know how to write to Redis (doesn't know how to serialize with Nippy).

I'm not familiar with the particular type here, but I suspect that it should be possible to coerce it to a standard Clojure data type before adding it to your session.

Does that make sense?

Edit to add: when you're using Ring's standard in-memory store, it can safely hold any kind of type since its values are all just JVM objects in memory. But when you switch to a persistent store like Carmine's store, you need to be mindful of the types being stored in sessions since there must be some way to serialize those types to/from Redis.

Carmine uses Nippy for its serialization, so can handle many different types - including every standard Clojure type. But Nippy will usually be unfamiliar with library-specific types (like your jdbc.next type here).

Pederaugust commented 8 months ago

Wow, thanks for the quick answer! I think it makes sense. {:id 1, :email , :username , :email_verified true} this is what I am trying to assoc to the session. Hmmm, so what I am trying to assoc is something related to jdbc, it looks like a clojure map, but I might be mistaken.

edit:

(do
            (println (-> (dissoc user :password_hash)
                         (dissoc :email_verification_token)
                         type))
            (assoc (redirect "/account") 
                   :session 
                   (assoc session 
                          :identity 
                          (-> (dissoc user :password_hash)
                              (dissoc :email_verification_token))))

Gives me:

clojure.lang.PersistentArrayMap

So maybe I need to read up on the nippy docs?

Pederaugust commented 8 months ago

Oh, so what I did was instead of dissocing, I just created an object with explicit keys and values. This solved the issue. So assume this was just a problem with the result received from next. Thanks for the guidelines! Keep up the good work!

ptaoussanis commented 8 months ago

I think it makes sense. {:id 1, :email , :username , :email_verified true} this is what I am trying to assoc to the session.

That shouldn't be a problem at all, Nippy happily supports clojure.lang.PersistentArrayMaps.

So maybe I need to read up on the nippy docs?

That shouldn't be necessary, that Carmine uses Nippy is mostly an implementation detail unless you're specifically wanting to do something advanced to take further advantage of specific Nippy capabilities.

In your case, it looks like some JDBC result is sneaking into your session somewhere. You'll need to find where that's happening.

A few places I'd start:

  1. Check the type of user in your code snippet above
  2. Check for possible Ring middleware that might be adding things to your Ring session without your realising
  3. Check for other code that writes sessions, it could be it's some other place in your code that's trying to add the JDBC result.
  4. Worst case, you could temporarily use a custom Ring store that'll throw debug info when it encounters the unexpected type. I can help with this if you need, just let me know.

Hope that helps!

ptaoussanis commented 8 months ago

Ah, great- so if removing the dissoc helped, then it's likely the issue (1.) I mentioned above - I suspect that user is some kind of jdbc.next type.

Even if Carmine is now satisfied, I'd suggest looking into the user type since it's conceivable that it could cause issues elsewhere.

This advice depends on specifics, but it's often a good idea to ensure that you're not leaking DB-specific types into your application logic.

Best of luck!