Surprising slowness in applyIndex

[jbfaden]

Ops.applyIndex is surprisingly slow in a case where Autoplot's getDataSets method is used to sort data read in. The sort of 80000 timetags is done in 30ms, but the applyIndex operator used to resort the data takes 15 seconds.

Here's an Autoplot script to demo:

from java.lang import System
from org.das2.qds import SortDataSet

ds= getDataSet('vap+cdaweb:ds=STA_L2_MAGPLASMA_1M&id=BFIELDRTN&timerange=2022-12-24+through+2022-12-31')

t0= System.currentTimeMillis()
s= sort(xtags(ds))
t1= System.currentTimeMillis()
print "sort in millis: ", t1-t0 
dsn= applyIndex(ds,s)
t2= System.currentTimeMillis()
print "applyIndex in millis: ", t2-t1 

dsn= SortDataSet(ds,s)
dsn= copy(dsn)
t3= System.currentTimeMillis()
print "manual application in millis: ", t3-t2

[jbfaden]

Note copy(SortDataSet(ds,s)) is fast and should be used to reimplement parts of applyIndex.

[jbfaden]

I've made it so that copy(SortDataSet(ds,s)) is used.

[jbfaden]

There's probably an issue, which would have been there before as well, where longs used to store TT2000s are converted to doubles. This should be checked on as well.

[jbfaden]

I experimented with some end-of-the-day time supporting Longs through this commonly-used operation. I'll start a new ticket for this: https://github.com/das-developers/das2java/issues/112