search

threatgrid / asami

A graph store for Clojure and ClojureScript
Eclipse Public License 1.0
634 stars 29 forks source link

Integer encoding is not always recognized #180

Closed quoll closed 3 years ago

quoll commented 3 years ago

Large data loads have given an error while trying to read an int. Stacktrace to come

quoll commented 3 years ago

Query:

(d/q '[:find (count ?a) . :where [?e ?a ?v]] db2)

Stack Trace:

   Unhandled java.lang.IllegalArgumentException
   Value out of range for int: -5368709120

                   RT.java: 1249  clojure.lang.RT/intCast
                   RT.java: 1219  clojure.lang.RT/intCast
             flat_file.clj:   27  asami.durable.flat-file/read-setup
             flat_file.clj:   25  asami.durable.flat-file/read-setup
             flat_file.clj:   75  asami.durable.flat-file.PagedFile/read_byte
               decoder.clj:  333  asami.durable.decoder/read-object-size
               decoder.clj:   -1  asami.durable.decoder/read-object-size
               decoder.clj:  352  asami.durable.decoder/read-object
               decoder.clj:   -1  asami.durable.decoder/read-object
             flat_file.clj:  174  asami.durable.flat-file.FlatFile/get_object
                 pool.cljc:   96  asami.durable.pool.DataPool/find_object
               common.cljc:   44  asami.durable.common$eval13793$fn__13794$G__13779__13797/invoke
               common.cljc:   44  asami.durable.common$eval13793$fn__13794$G__13778__13801/invoke
                  core.clj: 2628  clojure.core/partial/fn
                  core.clj: 6914  clojure.core/mapv/fn
             protocols.clj:  168  clojure.core.protocols/fn
             protocols.clj:  124  clojure.core.protocols/fn
             protocols.clj:   19  clojure.core.protocols/fn/G
             protocols.clj:   31  clojure.core.protocols/seq-reduce
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   13  clojure.core.protocols/fn/G
                  core.clj: 6830  clojure.core/reduce
                  core.clj: 6905  clojure.core/mapv
                  core.clj: 6905  clojure.core/mapv
             resolver.cljc:   57  asami.durable.resolver$eval16767$fn__16769$fn__16771/invoke
                  core.clj: 2759  clojure.core/map/fn
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                   RT.java:  535  clojure.lang.RT/seq
                  core.clj:  139  clojure.core/seq
                  core.clj:  139  clojure.core/seq
                query.cljc:  122  asami.query$eval11063$pattern_left_join__11068$fn__11069$iter__11070__11076$fn__11077$iter__11072__11078$fn__11079/invoke
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                   RT.java:  535  clojure.lang.RT/seq
                  core.clj:  139  clojure.core/seq
                  core.clj:  727  clojure.core/concat/fn
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                   RT.java:  535  clojure.lang.RT/seq
                  core.clj:  139  clojure.core/seq
                  core.clj: 2750  clojure.core/map/fn
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                   RT.java:  535  clojure.lang.RT/seq
                   RT.java:  982  clojure.lang.RT/nthFrom
                   RT.java:  940  clojure.lang.RT/nth
                  core.clj: 5037  clojure.core/distinct/step/fn/fn
                  core.clj: 5037  clojure.core/distinct/step/fn
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                 Cons.java:   39  clojure.lang.Cons/next
                   RT.java:  653  clojure.lang.RT/countFrom
                   RT.java:  643  clojure.lang.RT/count
                  core.clj:  882  clojure.core/count
                  core.clj:  876  clojure.core/count
                query.cljc:  737  asami.query$eval12199$aggregate_over__12204$fn__12205/invoke
                query.cljc:  699  asami.query$eval12199$aggregate_over__12204/invoke
                query.cljc:  819  asami.query$aggregate_query/invokeStatic
                query.cljc:  765  asami.query$aggregate_query/invoke
                query.cljc:  880  asami.query$eval12385$query_entry__12390$fn__12391/invoke
                query.cljc:  868  asami.query$eval12385$query_entry__12390/invoke
                 core.cljc:  239  asami.core$q/invokeStatic
                 core.cljc:  229  asami.core$q/doInvoke
               RestFn.java:  423  clojure.lang.RestFn/invoke
                      REPL:   33  user/eval19481
                      REPL:   33  user/eval19481
             Compiler.java: 7181  clojure.lang.Compiler/eval
             Compiler.java: 7136  clojure.lang.Compiler/eval
                  core.clj: 3202  clojure.core/eval
                  core.clj: 3198  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1977  clojure.core/with-bindings*
                  core.clj: 1977  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  829  java.lang.Thread/run

This is on a very large dataset built from a bundle in iroh-engine

quoll commented 3 years ago

Caused by booleans being decoded. The object pool used nil punning to determine when encoded data had not been returned, and went on to the data pool to read from the index. When a false value was returned, the nil punning assumed that nothing had been decoded and went to the index to look for the object at location 0xB000000000000000 (which is the encoded value of false)