search

scicloj / clojisr

Clojure speaks statistics - a bridge between Clojure to R
https://scicloj.github.io/clojisr/
Eclipse Public License 2.0
150 stars 10 forks source link

tech.ml.dataset do not recognize `NA` in logical columns #50

Closed daslu closed 4 years ago

daslu commented 4 years ago

Rserve has a maximal socket buffer size, and that limits the amount of data we can pass through it.

Our solution can be to wrap the r->clj and clj->r functions in a way that will transparently break the data into chunks before passing it over the socket.

genmeblog commented 4 years ago

I'm pretty sure they can. The error referenced in zulip discussion is a result of improper translation of logical columns

genmeblog commented 4 years ago

My conclusion is that transfer big amount of data is possible and works. What doesn't work is conversion of logical types to tech.ml.dataset

genmeblog commented 4 years ago

Next release of tech.ml.dataset will cover this issue

genmeblog commented 4 years ago

It's fixed already in t.m.ds