Revise sweepformula select operation

[MichaelHuth]

• [x] implementation
  • [x] add new tag to data to skip plotting in formula plotter, skip plotting filter.ranges component
  • [x] set data types on sub waves of combined data (multiple data sets of different type)
  • [x] in upmost wave add custom legend text representation of filter.ranges data
  • [x] workaround PSQ_RHEOBASE for sweepQC, ...Dashboard.ipf L342
  • [x] proper SCI handling (setQC only)
  • [x] add to data
  • [x] review play around
  • [x] add to other select dependent operations
  • [x] add second revision sel* operations
• [ ] docu
• [ ] Modifying ifn SF documentation needs to wait for https://github.com/AllenInstitute/MIES/pull/2023
• [ ] tests

close #2012

[MichaelHuth]

@t-b I think the SF changes could be technically tested.

Here are some examples:

swp = sweeps([1...10], 23, [30...40], 42)

sel1 = select(channels(AD), sweeps(1,2), selectVis(all), selectIVSCCsweepQC(passed))

sel2 = select(channels(), sweeps(), selectVis(all), selectIVSCCsweepQC(passed))

rng = selectRange(epochs("E1", $sel1))

ep = epochs("E1", $sel1)
lnb = labnotebook(ADC, [$sel1,$sel1])
sel = select($sel1, $sel1)

#tp = tp(tpbase(), [$sel1, $sel1])

sel3 = select($sel1, selectRange($ep + [0, 0]))

#data([$sel1, $sel2])
#data($sel1)
#$lnb

#data([select(), 10]) # fails

#data([select(), abc]) # fails

data([select(), $sel1])

Arrays for the select argument are allowed for tp, labnotebook, data. epochs takes only a single select argument. Since the argument order of select is now free, it was not possible to replace sweeps() with e.g. [1,2]. Thus, sweeps() takes now sweep numbers as arguments like sweeps(1, 2).

Rules:

• all new select*() operation can appear multiple times in select(...) except selectRange(...). selectRange(...) can appear only once.
• all select*() operations in select are logical ANDed (e.g. selectVis(all), selectVis(displayed) -> displayed)
• if select(...) has another select(...) as argument, then all unset filter properties are treated as 'allow all'
• if select(...) has no other select(...) as argument it uses the default settings as before, e.g. select()
• specifying selects as arrays for data means data(select()) is the same as data([select()]) and can be furhter extended with data([$sel0, $sel1, $sel2, ...]) where each select is treated independently with its attributed ranges.
• epochs takes only a single select argument (we can discuss if there is a use case for select arrays here)

The two expressions with the fails comment relate to the adapted formula executor.

[MichaelHuth]

I am currently looking into an issue where:

ep = epochs(E1, select())
selectRange($ep + [0, 0]) # works
selectRange(epochs(E1, select()) + [0, 0]) # fails parsing

pushed a preliminary fix

[t-b]

@timjarsky Ready for playing around.

[t-b]

Tim and I played around today. And it is very nice!

I have compiled a list of items to look at:

• [x] BSP_CheckProc_ChangedSetting(...)#L1293 [MIES_BrowserSettingsPanel.ipf] UpdateSweepPlot(...)#L6407 [MIES_MiesUtilities.ipf] SB_UpdateSweepPlot(...)#L357 [MIES_AnalysisBrowser_SweepBrowser.ipf] PostPlotTransformations(...)#L3970 [MIES_MiesUtilities.ipf] SF_Update(...)#L2764 [MIES_SweepFormula.ipf] PGC_SetAndActivateControl(...)#L231 [MIES_ProgrammaticGUIControl.ipf] SF_button_sweepFormula_display(...)#L2812 [MIES_SweepFormula.ipf] SF_ExecuteVariableAssignments(...)#L5990 [MIES_SweepFormula.ipf] SF_FormulaExecutor(...)#L1160 [MIES_SweepFormula.ipf] SF_OperationSelect(...)#L4925 [MIES_SweepFormula.ipf] JWN_SetStringInWaveNote(...)#L239 [MIES_JSONWaveNotes.ipf]

  
  Time: 2024-06-07T15:20:03+02:00
  Experiment: Basic (Packed)
  Igor Pro version: 9.0.6.1 (56657)
  ################################
  
  The line
  
  JWN_SetStringInWaveNote(selectResult, SF_META_DATATYPE, SF_DATATYPE_SELECT)
  
  in SF_OperationSelect needs to check if selectResult is null.

• [x] selectVis should have a default argument value of "displayed"

• [x] selectCM should have a default argument value of "all"

• [x] selectCM(all) does not select anything

• [x] Change prefix from sub-operations from select to sel to make it shorter

• [x] Prefix channels and sweeps with sel

• [x] Typo: sonly

• [x] SFH_IsValidSingleSelection: Please add a generic function which returns a wave with matches given a wildcard expresssion. There is nothing builtin into IP IIRC only stringmatch and ListMatch.

    for(matchSet : filter.stimsets)
        if(stringmatch(setName, matchSet))
            found = 1
            break
        endif
    endfor
    if(!found)
        return 0
    endif

note: The upper approach can already be done through FindIndizes with property wildcard matching. The utility function will return just true/false as the quoted code does.

• [x] SFH_IsValidSingleSelection: Can we only fetch the stimset name if required? For the standard case of match pattern * we could avoid that.
• [x] Allow selectRange() meaning the full range
• [x] select(channels(), channels(AD), ...) does not work as expected

Moved to issues:

• Does it make sense to move all operations accepting data operation as input to select arrays?
• Rename data operation to plot ?! Or better names?

[t-b]

@MichaelHuth CI will not run when you have merge conflicts.

[t-b]

@MichaelHuth How is the cursor operation handled by the changes here? Would you do selectRange(cursors(A, B))?

[t-b]

@timjarsky Does it make sense to add another selector selExp to select the experiment name? This would come in handy if you have sweeps from different experiments in one sweepbrowser. And what about selDev for selecting the device? And selSCI/selRAC?

[timjarsky]

Yes, those all make sense. Thanks @t-b

[MichaelHuth]

@MichaelHuth How is the cursor operation handled by the changes here? Would you do selectRange(cursors(A, B))?

cursors is not changed and returns an untyped array.

selRange returns datasets of ranges and has a datatype that allows it to be used with select as the datatype enables to have unordered select arguments. So selrange(cursors(A,B)) gets the cursor range. You can not use cursors as replacement for selrange. And you can only use two cursors in the argument of cursors inside selrange because selrange accepts only arrays with two elements (start / end).

Options would be:

• give cursors the same data type as selrange and change it to return a dataset. This would allow cursors to be used as replacement for selrange (with exactly two cursors specified).

[t-b]

So selrange(cursors(A,B)) gets the cursor range.

That's good enough.

[MichaelHuth]

Add wildcard support for selexp, seldev. Accept only a single match.

selsci:

• first get N x 3 wave with selections
• then use sweepNr, channelType, ChannelNumber with selsci to fill up missing SCI sweeps numbers (they have the same channelType/channelNumber)
• this feature is enabled as soon as selsci() is present, it takes no arguments

selrac: needs only sweep, so extend result from selsweeps right away

Regarding device and experiment:

• every sweep can originate from a specific device/experiment when loaded in the sweepbrowser
• useful function: PSX_GenerateComboKey L2955+

useful functions:

• AFH_GetSweepsFromSameSCI
• AFH_GetSweepsFromSameRACycle

[t-b]

I've tried playing around, but I get an assertion when loading data from the analysisbrowser:

PGC_SetAndActivateControl(...)#L231 [MIES_ProgrammaticGUIControl.ipf]
AB_ButtonProc_LoadSweeps(...)#L2749 [MIES_AnalysisBrowser.ipf]
PGC_SetAndActivateControl(...)#L231 [MIES_ProgrammaticGUIControl.ipf]
BSP_ButtonProc_ChangeSweep(...)#L1485 [MIES_BrowserSettingsPanel.ipf]
UpdateSweepPlot(...)#L6410 [MIES_MiesUtilities.ipf]
SB_UpdateSweepPlot(...)#L374 [MIES_AnalysisBrowser_SweepBrowser.ipf]
PostPlotTransformations(...)#L3978 [MIES_MiesUtilities.ipf]
LayoutGraph(...)#L2561 [MIES_MiesUtilities.ipf]
TUD_GetUserDataAsWave(...)#L177 [MIES_TraceUserData.ipf]
GetSetIntersection(...)#L3876 [MIES_Utilities.ipf]
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Time: 2024-06-26T13:26:17+02:00
  Experiment: Experiment (Packed)
  Igor Pro version: 9.0.6.1 (56657)
  ################################

The bug can also be reproduced with runwithOpts(testsuite = "UTF_TraceUserData.ipf") in Basic.pxp.

[MichaelHuth]

I've tried playing around, but I get an assertion when loading data from the analysisbrowser:

That was some fallout from the FindIndizes change to double precision as TUD also uses the combination with GetSetIntersection. I have fixed that.

I have added now selsci and selrac.

Currently the performance bottleneck is FindDimLabel is GetLastSetting. It is called very often due to SF_GetSelectData calling GetActiveChannels that iterates over all channels of the type.

[MichaelHuth]

Ready for another try.

I checked also the GetLastSetting DimLabel speed up and for the select case I tried it shows a 20% - 25% performance increase.

selsci now adds the sweeps of the same channeltype/channelnumber, it does not add e.g. the DAC channels of the headstage if through selchannels only ADC was selected. Does that make sense or should it also add the corresponding other channeltype / channelnumber (i.e. always ADC + DAC) ?

[t-b]

Does that make sense or should it also add the corresponding other channeltype / channelnumber (i.e. always ADC + DAC) ?

Yes that makes sense.

[t-b]

I checked also the GetLastSetting DimLabel speed up and for the select case I tried it shows a 20% - 25% performance increase.

Is that percent or percentage points?

[MichaelHuth]

I checked also the GetLastSetting DimLabel speed up and for the select case I tried it shows a 20% - 25% performance increase.

Is that percent or percentage points?

Results from my "test" in seconds:		FindDimlabel	FindValue
RAC	5.7	4.25
SCI	0.3	0.24

[t-b]

Todo:

• [x] Rename selSCI/selRAC to selExpandSCI/selExpandRAC
• [x] Add selSetCycleCount(n), selSetSweepCount(n): LBN "Set Sweep Count" and "Set Cycle Count"
• [x] Add a way to select the n'th SCI/RAC, do it with selSCIIndex(n) and selRACIndex(n)? Output range of valid sci/rac indizes on error message

selRACIndex:

• iterate over all loaded experiments
• get selections as is (sorted by sweep number)
• determine RAC id for each selection, find n-th different id, return all these selections

selSCIIndex: The index where scis have different id, not the index within an sci.

• iterate over all loaded experiments
• get selections as is (sorted by sweep number)
• get for each selection the headstage
• in a copy of the headstage wave, zapNaNs then uniques
• run over all unique headstages -> extract all selections with that headstage -> get SCI id for all these selections
• return all selections with n-th SCI id and concatenate
• repeat for next headstage

selSetCycleCount:

• counts sweeps that are repeated, is LNB entry: "Set Cycle Count"

selSetSweepCount:

• number of sweep in a stimset (stimset in WB can have multiple sweep waves) "Set Sweep Count"

selSCIIndex: SCI 0: Sweep 0, 1, 2 with headstage at AD0 SCI 1: Sweep 3, 4, 5 same headstage Variant 1: select(selchannels(AD0), selsweeps(0, 3), selSCIIndex(1)) selection: Sweep 0, AD0 (id0) selection: Sweep 3, AD0 (id1) combined with SCI index 1 there is one result, because we have Sweep 3 AD0 with a different SCI id.

Variant 2: we do an implicit SCI expansion select(selchannels(AD0), selsweeps(0, 3), selSCIIndex(1)) selection: Sweep 0, AD0 (id0) Sweep 3, AD0 (id1) combined with SCI index 1 there are three results, because expanding Sweep 3 AD0 adds Sweep 4, 5 and all of these have id1.

Variant 2 can be accomplished by using selExpandSCI(), thus we decide for variant 1.

Order:

1. selSetCycleCount, selSetSweepCount
2. selSCIIndex, selRACIndex
3. selExpandSCI, selExpandRAC