venantius / ultra

A Leiningen plugin for a superior development environment
Eclipse Public License 1.0
1.24k stars 35 forks source link

Fails with future returned by Datomic. #17

Closed benkamphaus closed 9 years ago

benkamphaus commented 9 years ago

Hi I've been playing around with ultra and enjoying it a lot. Unfortunately, most of the work I do at the repl is with Datomic and there's presently an issue with handling the deref of the future returned by a datomic transaction. If you include Datomic free in a library, this is a simple reproduce case:

(require '[datomic.api :as d])

(def db-uri "datomic:mem://test")
(d/create-database db-uri)
(def conn (d/connect db-uri))
@(d/transact conn [{:db/id (d/tempid :db.part/user) :db/doc "hello world"}])

Stacktrace:

java.lang.RuntimeException: Unable to convert: class datomic.btset.BTSet to Object[]
                                                                  ...
                                                clojure.core/to-array                core.clj:  333
                                                    clojure.core/sort                core.clj: 2828
                                                                  ...
                                                   clojure.core/apply                core.clj:  626
                                              clojure.core/partial/fn                core.clj: 2468
                                                                  ...
                                           puget.printer/sort-entries             printer.clj:  131
                                            puget.printer/eval2715/fn             printer.clj:  236
                                                                  ...
                               puget.printer/canonize-map/canonize-kv             printer.clj:  253
                                                  clojure.core/map/fn                core.clj: 2559
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/interleave/fn                core.clj: 3973
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                               clojure.core/drop/step                core.clj: 2646
                                                 clojure.core/drop/fn                core.clj: 2650
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                                  clojure.core/map/fn                core.clj: 2551
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                                   clojure.core/apply                core.clj:  624
                                                  clojure.core/mapcat                core.clj: 2586
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   28
                                                  clojure.core/map/fn                core.clj: 2557
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                                   clojure.core/apply                core.clj:  624
                                                  clojure.core/mapcat                core.clj: 2586
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   28
                                             fipp.printer/eval2406/fn             printer.clj:   67
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   31
                                                  clojure.core/map/fn                core.clj: 2557
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                                   clojure.core/apply                core.clj:  624
                                                  clojure.core/mapcat                core.clj: 2586
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   28
                                             fipp.printer/eval2394/fn             printer.clj:   55
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   31
                                                  clojure.core/map/fn                core.clj: 2557
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                                   clojure.core/apply                core.clj:  624
                                                  clojure.core/mapcat                core.clj: 2586
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   28
                                             fipp.printer/eval2377/fn             printer.clj:   44
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   31
                                                  clojure.core/map/fn                core.clj: 2557
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                                   clojure.core/apply                core.clj:  624
                                                  clojure.core/mapcat                core.clj: 2586
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   28
                                             fipp.printer/eval2377/fn             printer.clj:   44
                                                                  ...
                                               fipp.printer/serialize             printer.clj:   31
                                                  clojure.core/map/fn                core.clj: 2559
                                                                  ...
                                                    clojure.core/next                core.clj:   64
                                           clojure.core/concat/cat/fn                core.clj:  701
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                               clojure.core/concat/fn                core.clj:  685
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                                     clojure.core/seq                core.clj:  133
                                           clojure.core/concat/cat/fn                core.clj:  694
                                                                  ...
                                              clojure.core/chunk-next                core.clj:  667
                                            clojure.core.protocols/fn           protocols.clj:  101
                                          clojure.core.protocols/fn/G           protocols.clj:   19
                                    clojure.core.protocols/seq-reduce           protocols.clj:   31
                                            clojure.core.protocols/fn           protocols.clj:   54
                                          clojure.core.protocols/fn/G           protocols.clj:   13
                      clojure.core.reducers/reducer/reify/coll-reduce            reducers.clj:  112
                      clojure.core.reducers/reducer/reify/coll-reduce            reducers.clj:  112
                      clojure.core.reducers/reducer/reify/coll-reduce            reducers.clj:  112
                                                  clojure.core/reduce                core.clj: 6289
                                              transduce.reducers/each            reducers.clj:   44
                                      fipp.printer/pprint-document/fn             printer.clj:  215
                                         fipp.printer/pprint-document             printer.clj:  209
                                                 puget.printer/pprint             printer.clj:  389
                                          puget.printer/pprint-str/fn             printer.clj:  398
                                             puget.printer/pprint-str             printer.clj:  397
                                            whidbey.render/render-str              render.clj:   15
                                                                  ...
clojure.tools.nrepl.middleware.render-values/wrap-renderer/reify/send       render_values.clj:   35
     clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn/fn  interruptible_eval.clj:   79

With ultra disabled, the output of the deref is:

{:db-before datomic.db.Db@133188bf, :db-after datomic.db.Db@28cd0b35, :tx-data [#datom[13194139534312 50 #inst "2015-02-20T14:48:52.680-00:00" 13194139534312 true] #datom[17592186045417 62 "hello world" 13194139534312 true]], :tempids {-9223350046623220288 17592186045417}}
venantius commented 9 years ago

I believe this is essentially the same issue as #8 - I'm hoping to make some headway on both of these in the near future.

venantius commented 9 years ago

Hi @benkamphaus - what version of datomic free are you using? I can't seem to replicate your initial output.

venantius commented 9 years ago
user=> (require '[datomic.api :as d])
nil
user=> (def db-uri "datomic:mem://test")
#'user/db-uri
user=> (d/create-database db-uri)
true
user=> (def conn (d/connect db-uri))
#'user/conn
user=> @(d/transact conn [{:db/id (d/tempid :db.part/user) :db/doc "hello world"}])
{:t 1000}
benkamphaus commented 9 years ago

I see this behavior on the latest version (0.9.5130) on both free and pro, Lein dependencies as:

[com.datomic/datomic-free "0.9.5130"]

or

[com.datomic/datomic-pro "0.9.5130"]
venantius commented 9 years ago

@benkamphaus Would you mind seeing if this behavior is still present with the latest version of Ultra (0.3.0)? I believe this should be resolved.

venantius commented 9 years ago

Better make that 0.3.2, since I'm an idiot.

benkamphaus commented 9 years ago

I'm still seeing it with 0.3.2. I've observed the same behavior (as listed above) with:

java.lang.RuntimeException: Unable to convert: class datomic.btset.BTSet to Object[]

as the exception (stacktrace is identical).

Tested on:

Ubuntu 14.04 Mac OS X Yosemite

Environment isolated to this as only lein plugin.

JVMs:

Oracle JDK 1.7.0_67 Oracle JDK 1.7.0_60 Oracle JDK 1.8.0_31

Various Datomic versions tested, including:

0.9.5130 0.9.5067 0.9.4899

At least as far back as:

0.9.4572

Not sure if this info helps or is just superfluous.

venantius commented 9 years ago

What version of Clojure are you using?

benkamphaus commented 9 years ago

1.6.0

venantius commented 9 years ago

Great, and which version of Leiningen?

benkamphaus commented 9 years ago

2.4.2 on the Ubuntu VM. 2.5.0 on Yosemite.

venantius commented 9 years ago

Ah, great - I've been able to successfully reproduce this. Not quite sure how this sidestepped the fix introduced in 0.3.0, but I'll get to the bottom of it now.

venantius commented 9 years ago

@greglook I'm afraid this is boils down to another Puget issue; datomic.db.Db objects satisfy clojure.lang.IPersistentMap, but have custom print-methods defined. In this case the error is thrown because deep within the nested datomic.db.Db object is an object that Puget/Fipp can't render - but even if it could, Datomic's intention is not to print all of the information involved because the nested map is quite large.

It makes me wonder if, rather than testing for interface satisfaction, Puget should check for class membership, at least when it comes to printing documents that Puget expects to be core Clojure types.

I don't know if you have thoughts on this. I'm fine with building an escape hatch in Ultra for Datomic in the short term as well.

venantius commented 9 years ago

@benkamphaus - I've added a temporary escape hatch for datomic.db.Db objects in my development branch for 0.3.3 - it seems to resolve this issue. Would you mind taking a look at it? You can pull down a copy for installation here: https://github.com/venantius/ultra/tree/0.3.3

In the longer run I think @greglook and I will want to figure out something in Puget to address this, but I think this sort of escape hatch may be a necessary device in the near-to-mid term for Ultra to provide support for popular libraries like Datomic where I don't want to leave users out in the cold.

venantius commented 9 years ago

Hm. This escape hatch issue creates problems for projects that don't use Datomic. There's probably an easy way around this, too, but it warrants additional thinking.

greglook commented 9 years ago

It makes me wonder if, rather than testing for interface satisfaction, Puget should check for class membership, at least when it comes to printing documents that Puget expects to be core Clojure types.

Hmm, that might work, though it would be a bit more brittle that way. I think we're starting to bump into Puget's origins as a canonical serialization library rather than a colorizing pretty-printer.

I wonder if there's a way to check if a class is loaded without blowing up if it's not present? If so you could curate some format-doc extensions for Puget which only take effect when those classes are present.

venantius commented 9 years ago

That's sort of what I had in mind for my next task at this - grab the class, cast to string, check if that string matches a list of known class strings and if so, use the object's built-in print-method. Feels very monkey-patch-y, but it's also relatively painless to backdoor into Puget from Ultra.

venantius commented 9 years ago

The latest commit (https://github.com/venantius/ultra/commit/2fded317b8f150161a5855df61be81a520945c9e) on the 0.3.3 branch covers this.

venantius commented 9 years ago

Closing as this will be resolved once 0.3.3 ships.

venantius commented 9 years ago

@benkamphaus 0.3.3 is up on Clojars now; datomic should be happy working with it. Let me know if you find this not to be the case.

benkamphaus commented 9 years ago

Works beautifully! Thanks!

<clj> [user]$  @(d/transact conn [{:db/id (d/tempid :db.part/user) :db/doc "hello world"}])
{:db-after datomic.db.Db@5fbf819c
 :db-before datomic.db.Db@1eb71cfe
 :tempids {-9223350046623220288 17592186045417}
 :tx-data [#datom[13194139534312 50 #inst "2015-03-16T17:13:57.415-00:00" 13194139534312 true] #datom[17592186045417 62 "hello world" 13194139534312 true]]}