search

das-developers / das2java

The original das2 library. Provides interactive publication-ready 2-D plotting
https://das2.org
GNU Lesser General Public License v3.0
4 stars 0 forks source link

Surprising slowness in applyIndex #101

Open jbfaden opened 3 months ago

jbfaden commented 3 months ago

Ops.applyIndex is surprisingly slow in a case where Autoplot's getDataSets method is used to sort data read in. The sort of 80000 timetags is done in 30ms, but the applyIndex operator used to resort the data takes 15 seconds.

Here's an Autoplot script to demo: from java.lang import System from org.das2.qds import SortDataSet

ds= getDataSet('vap+cdaweb:ds=STA_L2_MAGPLASMA_1M&id=BFIELDRTN&timerange=2022-12-24+through+2022-12-31')

t0= System.currentTimeMillis() s= sort(xtags(ds)) t1= System.currentTimeMillis() print "sort in millis: ", t1-t0 dsn= applyIndex(ds,s) t2= System.currentTimeMillis() print "applyIndex in millis: ", t2-t1

dsn= SortDataSet(ds,s) dsn= copy(dsn) t3= System.currentTimeMillis() print "manual application in millis: ", t3-t2

jbfaden commented 3 months ago

Note copy(SortDataSet(ds,s)) is fast and should be used to reimplement parts of applyIndex.