Microsecond resolution for valid-time?

[stevana]

Is there any way to get microsecond resolution for valid-time?

When I do:

(crux/submit-tx
 db
 [[:crux.tx/put
   {:crux.db/id :foo
    :message "msg1"}
   (clojure.instant/read-instant-timestamp "2020-05-27T10:10:52.494217+00:00")]
  [:crux.tx/put
   {:crux.db/id :foo
    :message "msg2"}
   (clojure.instant/read-instant-timestamp "2020-05-27T10:10:52.494279+00:00")]])

(crux/entity-history (crux/db db) :foo :asc
                     {:with-docs? true, :with-corrections? true})

I only see "msg2", presumably because "msg1" and "msg2" have the same valid-time when restricted to milliseconds?

[refset]

Hi @stevana we don't natively support higher-resolution timestamps just yet, although we have been thinking about making this configurable eventually, to open up support for nanoTime() nanoseconds. Whilst there is no immediate roadmap agenda for this work we would be keen to hear more about your use-case/requirements - we could potentially accelerate a decision and implementation.

Right now we rely on 64-bit Java Dates that represent valid-time and transaction-time as milliseconds since 1970. As it happens, I have recently played around with manipulating these 64 bit values purely in userspace to encode entirely different time ranges, including vector clock information, so I know it's already possible to support microseconds if you are willing to use a few "translation" functions on the way in & out of the Crux tx operations and db+history APIs. The downside of encoding microseconds like this is that you will need to constrain the range of expressible timestamps at the other end of the scale, although I doubt your use-case will make use of those bits anyway :)

This encoding/decoding experiment I wrote for a Hybrid Logical Clock (as in https://jaredforsyth.com/posts/hybrid-logical-clocks/) sort of illustrates what I mean:

(def ^:const total-bsize 64)

(def ^:const ms-bsize 42) ;; (= (Date. max-ms) #inst "2109-05-15T07:35:11.103-00:00")
;; min time supported 1970-01-01
;; max time supported #inst "2109-05-15T07:35:11.103-00:00"
(def ^:const count+node-bsize (- total-bsize ms-bsize)) ;; 22b

(def ^:const count-bsize 10) ;; max count 1024 ;; max nodes 4096
(def ^:const node-bsize (- count+node-bsize count-bsize))

(def ^:const not-count-bsize (- total-bsize count-bsize))
(def ^:const not-node-bsize (- total-bsize node-bsize))

(def ^:const max-ms (dec (bit-set 0 ms-bsize)))
(def ^:const max-count (dec (bit-set 0 count-bsize)))
(def ^:const max-node (dec (bit-set 0 node-bsize)))

(defn- hlc->long [{:keys [ms count node]}]
  {:pre [(<= 0 ms)
         (<= ms max-ms)
         (<= count max-count)
         (<= node max-node)]}
  (bit-or node
          (bit-shift-left count node-bsize)
          (bit-shift-left ms count+node-bsize)))

(defn- long->hlc [l]
  {:ms (unsigned-bit-shift-right l count+node-bsize)
   :count (-> (bit-shift-left l ms-bsize)
                (unsigned-bit-shift-right not-count-bsize))
   :node (-> (bit-shift-left l not-node-bsize)
             (unsigned-bit-shift-right not-node-bsize))})

(defn hlc-long->date [l]
  (Date. (:ms (long->hlc l))))

;; (long->hlc (hlc->long {:ms (inst-ms (Date.)) :count 13 :node 1})) ;; test

[stevana]

I would like to create a database of testing outcomes, for Jepsen-like tests (model-based system tests, with inherent flakiness).

The use case I had in mind was to insert each line of a log file with a different valid-time, and be able to get back the whole log by looking at the entire history. Perhaps be able to merge different logs to get a interleaved/merged view in chronological order, get the test results for all tests which have logs containing some error message, etc.

I would have thought that using Java 8's java.time for valid-time would do the trick, but perhaps there are other complications in the implementation that I'm not aware of?

I'll have a look at your suggestion of encoding/decoding other time ranges, cheers!