search

scicloj / wolframite

An interface between Clojure and Wolfram Language (the language of Mathematica)
Mozilla Public License 2.0
53 stars 2 forks source link

Wolfram Engineering: Avoid transferring/converting huge data when not explicitly requested #119

Open holyjak opened 1 month ago

holyjak commented 1 month ago

Some Wolfram functions such as Import of a 400k CSV or VideFrameList may return huge data, and transferring that and parsing into JLink Expr and than Clojure is slow typically not really desired. In most cases you want to store them into a Wolfram-side variable and not transfer them back.

It would be good to notice that we are trying to transfer huge data and abort / shorten unless explicitly instructed not to. Similarly as println does when *print-length* / depth is set.

The big question is how do we detect this case... Not sure whether it is possible on the JVM side, without going through all the data. Is there perhaps some Wolfram fn that can tell us the "size" of the result, which we could call automatically (or for a list of "dangerous" functions) to find out the size before we start transferring all the data? Or is there some other way?

light-matters commented 1 month ago

An example of generating a lot of data is 

  (-> path--video
      w/Video
      (w/VideoTrim [12 24])
     (w/VideoFrameList 50)