techascent / tech.ml.dataset

A Clojure high performance data processing system
Eclipse Public License 1.0
680 stars 35 forks source link

transit-json WRITE - roaringbitmap exception #425

Open awb99 opened 1 month ago

awb99 commented 1 month ago

I am saving datasets as transit-json. When I create a dataset with a zoned-date-time column and then convert the type to instant and try to save this dataset as transit-json, then I get an exception. Below is the a namespace to reproduce the problem. The precise exception is logged below as well.

I use latest version of tml ds via latest version of tablecloth.

(ns demo.experiment
  (:require
   [tick.core :as t]
   [tech.v3.dataset :as ds]
   [tablecloth.api :as tc]
   [tech.v3.libs.clj-transit :as tech-transit]
   [tech.v3.io :as io]))

(defn t->file
  [ds fname]
  (with-open [outs (io/output-stream! fname)]
    (tech-transit/dataset->transit ds outs)))

(defn sanitize-date [ds]
  (tc/convert-types ds {:date [:instant #(t/instant %)]}))

(-> (tc/dataset {:date [(t/zoned-date-time)]
                 :a [1]
                 :b [2.0]})
    (sanitize-date)
    (t->file "zoned.transit-json")
    )
;; => Execution error at com.cognitect.transit.impl.AbstractEmitter/marshal (AbstractEmitter.java:195).
;;    Not supported: class org.roaringbitmap.RoaringBitmap