flucoma / flucoma-sc

Fluid Corpus Manipulation plugins for Supercollider
BSD 3-Clause "New" or "Revised" License
70 stars 16 forks source link

change the JSON parser to the native one #142

Open tremblap opened 1 year ago

tremblap commented 1 year ago

as disucssed with @weefuzzy we had a pre 3.6 home-made parser. we could implement it as a fall back with try like in FluidSound, or just remove it and replace by the following code which is much faster for large DS to dump to dict for instance (via the tempfile)

(
~parse = {
    var str = File.readAllString("zepath");
    var d = str.parseJSON; 
    d
};
~dic = ~parse.value; 
~dic.postln; 
)
weefuzzy commented 1 year ago

Looks like it was changed to the handrolled thing in c1f7725d71748d9c3807f0d8ed607adf78f4f31b because the native parsing was enforcing a change of types somewhere undesirable.

weefuzzy commented 1 year ago

@g-roma do you happen to remember what parseYAMLFie was doing that was undesirable?

g-roma commented 1 year ago

I can't remember, but there was some problem.

weefuzzy commented 1 year ago

The problem seems to be that parseYAML and friends make everything a String, e.g.

d = "{\"foo\":[1.0,2.0,3,4,5]}".parseJSON
d["foo"].do{|x| x.class.postln}
-> Dictionary[ (foo -> [ 1.0, 2.0, 3, 4, 5 ]) ]
String
String
String
String
String

🤦

So, I guess the alternative to what we're doing now is to walk the whole Stringified Dictionary after faster parsing and coerce values back to float where appropriate. That may or may not be faster than the current implementation.

weefuzzy commented 1 year ago

see https://github.com/supercollider/supercollider/issues/1154 which has been open since 2014

looks like yaml-cpp doesn't provide a nice way to determine whether a scalar is a string, int or float (see https://stackoverflow.com/questions/19994312/obtain-type-of-value-stored-in-yamlnode-for-yaml-cpp)

tremblap commented 1 year ago

Indeed @weefuzzy at the moment I can read faster but I cannot use any of it without converting to float the strings... not super useful :) Slow it stays then, until we find a way around this.

tremblap commented 1 year ago

actually, running

i["cols"] = i["cols"].asInteger;

)

(
x = Main.elapsedTime;
i["data"].keysValuesDo{|a,b|i["data"][a] = b.asFloat};
(Main.elapsedTime - x).postln;
)

takes 100% cpu for quite some time. So I don't know what the answer to

That may or may not be faster than the current implementation.

is yet...

tremblap commented 1 year ago

ok our version is faster than parsing the dict to itself - although it suddenly occured to me that maybe I'm making an infinite loop by parsing over itself - another test is needed.