asami

[borkdude]

I just tried out asami locally:

https://github.com/threatgrid/asami

$ ./asami
(["Explorers"] ["Demolition Man"] ["Johnny Mnemonic"] ["Toy Story"])

Worked great, memory consumption during compilation is far more reasonable than datascript. Hopefully once it gets durable storage it keeps working well like this.

[BrunoBonacci]

Great, when you get chance could you please push an example?

[quoll]

Asami would be useful as a repl-driven app, so I tried this...

(ns asami.main
  (:require [clojure.main :as cm]
            [clojure.repl :as repl]
            [asami.core :as asami])
  (:gen-class))

(defn -main
  [& args]
  (require '[asami.core :as asami :refer [now instant instant? long-time create-database connect
                                          db as-of as-of-t since since-t graph as-connection
                                          transact entity q show-plan import-data export-data]])
  (apply cm/main args))

This works well with lein jar. Unfortunately, after building it gets into a loop of printing a repl prompt and an exception:

user=> Execution error (IllegalArgumentException) at asami.main/-main (main.clj:20).
No matching field found: read for class clojure.lang.LineNumberingPushbackReader

I thought that maybe the .read was being picked up via the Reader interface, or even through reflection, and hence the linker dropped the function... or something like that. But creating a LineNumberingPushbackReader around a StringReader and then reading from it works fine, though the repl code still complains.

Do you have any suggestions please?

[borkdude]

@quoll

With "after compilation" do you mean graalvm native-image compilation?

In this case don't require at runtime, rather pull the require to the top-level.

Can you give an example of how you would call your jar / native binary?

I think it's probably better to make the operations of your application explicit rather than going through clojure.main, but this depends on your example.

[borkdude]

@quoll Here is an example that demonstrates that asami works with native image compilation (well, this specific example):

https://github.com/borkdude/jayfu/commit/7e6101c298ae5500e94133a8ce1a75591a745b27

I adapted the jayfu tutorial.

$ ./jayfu
(["Explorers"] ["Demolition Man"] ["Johnny Mnemonic"] ["Toy Story"])

[borkdude]

When using local storage ("asami:local://dbname") hits a reflection issue:

$ ./jayfu
Exception in thread "main" java.lang.IllegalArgumentException: No matching field found: getTime for class java.util.Date
    at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
    at asami.internal$long_time.invokeStatic(internal.cljc:26)
    at asami.internal$long_time.invoke(internal.cljc:23)
    at asami.durable.store$fn__10772$new_db__10777$fn__10778.invoke(store.cljc:63)
    at asami.durable.store$fn__10772$new_db__10777.invoke(store.cljc:61)
    at asami.durable.store$fn__11150$create_database__11155$fn__11156.invoke(store.cljc:254)
    at asami.durable.store$fn__11150$create_database__11155.invoke(store.cljc:247)
    at asami.core$connection_for.invokeStatic(core.cljc:39)
    at asami.core$connection_for.invoke(core.cljc:32)
    at asami.core$fn__11800$create_database__11805$fn__11806.invoke(core.cljc:57)
    at asami.core$fn__11800$create_database__11805.invoke(core.cljc:51)
    at jayfu.main$_main.invokeStatic(main.clj:17)
    at jayfu.main$_main.doInvoke(main.clj:15)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at jayfu.main.main(Unknown Source)

[borkdude]

The above issue should be fixed with https://github.com/threatgrid/asami/pull/156

[borkdude]

New issue:

$ ./jayfu
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No matching field found: close for class sun.nio.ch.FileLockImpl
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at clojure.core$deref_future.invokeStatic(core.clj:2304)
    at clojure.core$deref.invokeStatic(core.clj:2324)
    at clojure.core$deref.invoke(core.clj:2310)
    at jayfu.main$_main.invokeStatic(main.clj:16)
    at jayfu.main$_main.doInvoke(main.clj:15)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at jayfu.main.main(Unknown Source)
Caused by: java.lang.IllegalArgumentException: No matching field found: close for class sun.nio.ch.FileLockImpl
    at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
    at asami.durable.store$fn__11015$transact_update_STAR___11020$fn__11024.invoke(store.cljc:187)
    at asami.durable.store$fn__11015$transact_update_STAR___11020.invoke(store.cljc:178)
    at asami.durable.store$fn__11050$transact_data_STAR___11059$fn__11064.invoke(store.cljc:217)
    at asami.durable.store$fn__11050$transact_data_STAR___11059.invoke(store.cljc:208)
    at asami.durable.store.DurableConnection.transact_data(store.cljc:232)
    at asami.core$fn__11938$transact__11943$fn__11950$fn__11958.invoke(core.cljc:187)
    at asami.core$fn__11938$transact__11943$fn$reify__11977.get(core.cljc:200)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:829)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

[borkdude]

After fixing those reflection issues in asami and adding an additional reflection config for java.lang.Boolean (@quoll knows why, this was needed because booleans were serialized using some reflection I think), it works with local storage!

$ ./jayfu
(["Explorers"] ["Demolition Man"] ["Johnny Mnemonic"] ["Toy Story"])

See: https://github.com/borkdude/jayfu/tree/asami

[borkdude]

After multiple runs, a new reflection issue:

$ ./jayfu
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No matching field found: id for class asami.graph.InternalNode
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at clojure.core$deref_future.invokeStatic(core.clj:2304)
    at clojure.core$deref.invokeStatic(core.clj:2324)
    at clojure.core$deref.invoke(core.clj:2310)
    at jayfu.main$_main.invokeStatic(main.clj:16)
    at jayfu.main$_main.doInvoke(main.clj:15)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at jayfu.main.main(Unknown Source)
Caused by: java.lang.IllegalArgumentException: No matching field found: id for class asami.graph.InternalNode
    at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
    at asami.graph.InternalNode.equals(graph.cljc:43)
    at clojure.lang.Util.equiv(Util.java:33)
    at clojure.lang.PersistentHashMap$BitmapIndexedNode.find(PersistentHashMap.java:788)
    at clojure.lang.PersistentHashMap.containsKey(PersistentHashMap.java:126)
    at clojure.lang.APersistentSet.contains(APersistentSet.java:34)
    at clojure.lang.PersistentHashSet.cons(PersistentHashSet.java:97)
    at clojure.lang.PersistentHashSet.cons(PersistentHashSet.java:17)
    at clojure.lang.RT.conj(RT.java:677)
    at clojure.core$conj__5407.invokeStatic(core.clj:87)
    at clojure.core$conj__5407.invoke(core.clj:84)
    at zuko.entity.writer$fn__11355$map__GT_triples__11360$fn__11361$fn__11369.invoke(writer.cljc:167)
    at zuko.entity.writer$fn__11355$map__GT_triples__11360$fn__11361.invoke(writer.cljc:166)
    at zuko.entity.writer$fn__11355$map__GT_triples__11360.invoke(writer.cljc:153)
    at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422$fn__11427.invoke(writer.cljc:197)
    at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422.invoke(writer.cljc:181)
    at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422$fn__11425.invoke(writer.cljc:194)
    at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422.invoke(writer.cljc:181)
    at asami.entities$fn__11638$entity_triples__11643$fn__11647.invoke(entities.cljc:111)
    at asami.entities$fn__11638$entity_triples__11643.invoke(entities.cljc:44)
    at asami.entities$fn__11714$build_triples__11719$fn__11720$add_triples__11730.invoke(entities.cljc:147)
    at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
    at clojure.core$reduce.invokeStatic(core.clj:6829)
    at clojure.core$reduce.invoke(core.clj:6812)
    at asami.entities$fn__11714$build_triples__11719$fn__11720.invoke(entities.cljc:163)
    at asami.entities$fn__11714$build_triples__11719.invoke(entities.cljc:135)
    at asami.core$fn__11938$transact__11943$fn__11950$fn__11958$fn__11973.invoke(core.cljc:191)
    at asami.durable.store$fn__11050$transact_data_STAR___11059$fn__11064$fn__11065.invoke(store.cljc:219)
    at asami.durable.store$fn__11015$transact_update_STAR___11020$fn__11024.invoke(store.cljc:193)
    at asami.durable.store$fn__11015$transact_update_STAR___11020.invoke(store.cljc:178)
    at asami.durable.store$fn__11050$transact_data_STAR___11059$fn__11064.invoke(store.cljc:217)
    at asami.durable.store$fn__11050$transact_data_STAR___11059.invoke(store.cljc:208)
    at asami.durable.store.DurableConnection.transact_data(store.cljc:232)
    at asami.core$fn__11938$transact__11943$fn__11950$fn__11958.invoke(core.cljc:187)
    at asami.core$fn__11938$transact__11943$fn$reify__11977.get(core.cljc:200)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:829)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

[borkdude]

The binary works again with PR https://github.com/threatgrid/asami/pull/157 (which supersedes PR 156).

Note that there are some remaining reflections in zuko/logging.clj but this isn't hit in the above example.

$ clj -M -e "(set! *warn-on-reflection* true)" -e "(require 'asami.core)"
true
Reflection warning, zuko/logging.clj:14:36 - reference to field getClassName can't be resolved.
Reflection warning, zuko/logging.clj:13:30 - reference to field getStackTrace can't be resolved.
Reflection warning, zuko/logging.clj:78:5 - call to method append on java.io.Writer can't be resolved (argument types: unknown).
planner.cljc:107 recur arg for primitive local: mcount is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: mcount
Reflection warning, asami/durable/block/file/voodoo.clj:18:14 - reference to field clean on java.lang.Object can't be resolved.

[quoll]

@quoll

With "after compilation" do you mean graalvm native-image compilation?

In this case don't require at runtime, rather pull the require to the top-level.

Can you give an example of how you would call your jar / native binary?

I think it's probably better to make the operations of your application explicit rather than going through clojure.main, but this depends on your example.

We discussed this, and it’s embarrassingly obvious why a plain repl could never work for this.

But I’ll just explain the require: it was a hack to make the namespace alias and :refer symbols available to the repl. This does not happen when done at the top level.