search

openworm / tracker-commons

Compilation of information and code bases related to open-source trackers for C. elegans
11 stars 12 forks source link

Python wcon pip possible performance issues #170

Open cheelee opened 5 years ago

cheelee commented 5 years ago

It took my machine 3 and a half minutes to load a 48Mb (after unzip) file - https://zenodo.org/record/1034927/files/unc-80%20(e1069)%20nRHO-1%20QT309%20on%20food%20L_2011_11_11__13_12___3___4.wcon.zip

It takes only a few seconds on the partial javascript implementation at http://movement.openworm.org/wconviewer to load the same file.

I think we need to figure out if our common use-cases demand such a heavy-weight implementation, and if there is a way to allow the common case to run a lot quicker.

Ichoran commented 5 years ago

Isn't this mostly due to schema checking? IIRC that was most of the overhead. (The Scala was still way faster--maybe half a second for 48MB if that, but they weren't absurdly far apart any longer.)

MichaelCurrie commented 5 years ago

Yes it's because the library implementing the schema checking that we are relying on is painfully slow for some reason.

I even raised this as an issue on their repository:

https://github.com/Julian/jsonschema/issues/277

So this issue was sort of already closed as a #wontfix, since the workaround is just to use the option not to check the schema if you want speed.

https://github.com/openworm/tracker-commons/issues/55

cheelee commented 5 years ago

Ah! Thanks for the info! I'll see what I can do about it, and perhaps offer an option to enable/disable schema-checking depending on what we think ought to be the default for WCON file loads.