Allow the user to do basic analysis on sweeps

[t-b]

User requirements:

• Select one x range per channel type (DA, AD, TTL) using cursor pairs
• View the results of Wavestats (mean, stddev, min, max), using either all properties, or selected ones, in a table where the results are displayed by channel/sweep and grouped by channel type
• Allow to plot different WaveStats properties against each other or against the sweep numbers
• We never mix data from different sweeps (for now)

Implementation:

• Add a new tab to the Databrowser/Sweepbrowser
• Use the current infrastructure for gathering trace infos -> PA_GetTraceInfos
• Create separate graphs for each plot, multiple graphs are already see "Export Traces"
• Table data and plots update automagically when the cursors are changed
• The created graph should have the formula shown so that the user can replicate it.
• Overlay sweeps

We serialize the GUI selections

into formulas:

(Mean(Derivative((Cursors(AB), AD, "1;3")))-Mean([100ms, 200ms], AD)) vs (Max(Cursors(EF), DA) + Stddev(Cursors(GH), DA))

Where "1;3" is the range of sweeps, defaulting to all sweeps. The sweep list can be also include "removed headstages". The syntax is "1[0,2];2;" which means that headstage 0 and 2 are removed from sweep 1 and all headstages are shown for sweep 2. Sweeps 1 and 3 can have different x scaling and different number of points.

Further ideas:

• Seralize these formulas to a JSON file on disc, set the analysis type in the wavebuilder to a stimset and do the analysis automatically.

Known limitations:

• First version will only work with the databrowser as don't handle duplicated sweep numbers.

Data magnitudes:

• Up to 100 sweeps
• Up to 2e6 points per sweep
• At most 8 headstages per sweep

Related issue #38.

[ukos-git]

Formula Parser

tokenizer

• start token (
  • function name
  • parse parameters
  • token , next parameter
  • token ( --> recurse
  • end token )

parse formula to structured json

respect order of calculation: * > +

min - max * sdev

[
  {
    "operator": "+",
    "operation": [
      {
        "function": "min",
        "range": [
          "-inf",
          "inf"
        ],
        "channelType": "AD",
        "sweepList": "1;2;3;"
      }
    ]
  },
  {
    "operator": "-",
    "operation": [
      {
        "operator": "+",
        "operation": [
          {
            "function": "max",
            "range": [
              "-inf",
              "inf"
            ],
            "channelType": "AD",
            "sweepList": "1;2;3;"
          }
        ]
      },
      {
        "operator": "*",
        "operation": [
          {
            "function": "sdev",
            "range": [
              "-inf",
              "inf"
            ],
            "channelType": "AD",
            "sweepList": "1;2;3;"
          }
        ]
      }
    ]
  }
]

Parameters

• function name: Mean, Derivative, Min, Max
• range [start, end] (default to [-inf, inf]
• channelType (default from cursor)
• sweepList = "1;1[0,2];2;3[1]" with headstage removal in square brackets

Formula Check

check valid function names

define a list of valid functions like min,max, derivative

check input wave type

3 types of input waves:

• multiple point sweep calculations (size(sweeplist), arbitrary length numpnts(sweep)) → WAVE of /WAVEs to account for different sweep sizes. return modified input wave
• one point functions (size(sweeplist), 1) storing the result of wavestats with size 1
• Cursor functions (2, 1) returns [csr(N), csr(M)] for Cursor(NM)

FAIL for invalid input wave usage:

Mean(Derivative) →ok Derivative(Mean) →fail

operations possible on same type and mixed type input waves:

• (min + max)
• (min * derivative)
• (derivative * derivative)

[t-b]

@timjarsky We've discussed how to tackle that today. We would leverage the new JSON XOP to use JSON as intermediate data structure for the parsed formula. This does require IP8 but we already agreed to switch to IP8 only IIRC.

[t-b]

@timjarsky Thx.

[ukos-git]

I consider using a variant of json logic for operations:

{
  "+": [
    1,
    1
  ]
}

instead of the previous suggestion which would look more complicated:

{
  "operator": "+",
  "operation": [
    1,
    1
  ]
}

[ukos-git]

so the syntax for a function call would be similar with the function name as group member.

{
  "min": {
    "range": [
      "-inf",
      "inf"
    ],
    "channelType": "AD",
    "sweepList": "1;2;3;"
  }
}

[t-b]

Don't forget the ignored headstages which probably means that sweeplist needs to be split up.

[ukos-git]

Other than json logic, I currently do not support more than 2 columns of operations:

{
  "+": [
    1,
    1,
    1
  ]
}

That would make the parsing more complicated and I don't see the benefits, as we would need to create the equivivalent wrappers to MatrixOp sumCols for all other operations like columnwise multiplication etc.

You can still add formula like 1+1+1 but it will be calculated in two steps as 1+(1+1).

{
  "+": [
    1,
    {
      "+": [
        1,
        1
      ]
    }
  ]
}

[timjarsky]

@ukos-git is it also possible to support channel number? e.g. AD2

[t-b]

@timjarsky channel numbers are not in the original proposal but we can certainly add them.

@ukos-git we have specified "ad" as all channels of type ad, now "ad2" would mean the obvious.

[ukos-git]

Other than json logic, I currently do not support more than 2 columns of operations:

The multi-column approach is back for addition and multiplication with the current state machine code.

[ukos-git]

multi-columns now for all operations. For the inverse operations (-, /) the first element (dividend, minuend) is inverted. State Machine now fully working but uses lookaheads to determine the order of calculation.

[timjarsky]

@ukos-git @t-b regarding

operations possible on same type and mixed type input waves:

(min + max)
(min * derivative)
(derivative * derivative)

could we also support (in this PR or in the future) e.g., max(derivative(AD)) and max(derivative(derivative(AD)) and integral variations?

[ukos-git]

concatenation of functions should not be a problem. I hope the formula parser is robust enough.

max(derivative(derivative(sweeps(cursors(A,B),channel(AD),0;1;2))))

There are still some issues. The most urgent is that operations with sweeps like addition have to be reworked in another way. Calculating with 1dimensional arrays works, though, which means that you can use min(sweeps(cursors(A,B),channel(AD),0;1;2))/2 but not sweeps(cursors(A,B),channel(AD),0;1;2)/2

• [x] operations with 2d arrays
• [x] operations between 2d and 1d arrays, see https://github.com/AllenInstitute/MIES/issues/282
• [x] operations between 2d arrays and constants
• [x] operations between 1d arrays and constants

Other minor issues are

• [x] Add Integrate function
• [x] allow white space in formula
• [x] count correct bracket usages
• [x] Automatically Add Cursors
• [x] Add Cursor hooks, https://github.com/AllenInstitute/MIES/issues/283
• [x] Handle non-existing sweeps returns (currently a null-waveref is returned which causes issues.
• [x] Better Error Handling. Can we catch the ASSERTs somehow during parsing and execution? Separate issue https://github.com/AllenInstitute/MIES/issues/275.

JSON Logic has some additional elements that could come in useful: http://jsonlogic.com/operations.html Please take a look and see if there is anything interesting that could help.

I would personally add an array creation operation (…) for an easier sweepList and sweepList should be an array then.

1…5 → [1,2,3,4,5]

[timjarsky]

@t-b if you have time tomorrow (Friday) could we test this together?

[t-b]

@timjarsky Yes I have time, how about 10am?

[timjarsky]

@t-b 10am will work. FYI the display button may be missing it's control action procedure.

[t-b]

@ukos-git Here is the whishlist of changes from the testing on friday.

Roughly in the order of importance:

• [x] Error out on trying to use sweeps from different experiments (later addition perhaps)
• [x] What happens if a specified sweep in the formula is not displayed? Do we error out, if not can we just add it using OVS and continue? (OVS_ChangeSweepSelectionState), see https://github.com/AllenInstitute/MIES/issues/271
• [x] One trace per channel number (i.e. if you have AD0 and AD9 shown in the data browser, we want to have two traces)
• [x] Allow AD0 (with channel number) for channel selection
• [x] one channel on one side -> plot each trace of the other side against that one channel. (aka All-To-One instead of the usual All-To-All). (separate issue)
• [x] existing graphs are not handled properly (bug)
• [x] explicit range (instead of cursors) with [xxx ms, yyy ms] does not work (already implemented without units)
• [x] Convert sweep range ("1;2;3") to an array with ... array creation support. Instead of [1, 2, 3, 4] one can write [1...4]. Allow for [] to mean all displayed sweeps, one can also use [0...inf] to mean all available sweeps. (separate issue)
• [x] Reevaluate formula and update graph when new data arrives and "Display Next Sweep" is checked. (DB_UpdateToLastSweep is the entry point after acquiring and saving the sweep)
• [x] Add rms and avg as operations
• [x] add support for multiline formulas (notebook in GUI), formulas should ignore all whitespace (except EOLs for comments), # is the comment char (ignores everything until the end of the line). Use NormalizeToEOL as well to cope with all EOLs.
• [x] Implement area instead of integrate, separate issue https://github.com/AllenInstitute/MIES/issues/269
• [x] Add Butterworth operation having (..., lowPassCutoff, highPassCutoff, order) parameters, see Wavebuilder and its use of FilterIIR
• [x] Add automatic graph axis labels -> e.g. min($unit), min(a.u.) if the units are not matching, issue https://github.com/AllenInstitute/MIES/issues/272
• [x] Add time operation which returns the x-values of its input

NeedsMoreInfoTopics

• [x] Add button to store the formula in as JSON file on disc. The idea is to be able to load it via the analysis browser and execute the analysis on multiple experiments.

Example:

  # save to JSON
  [
    "description" : "minimum",
    "formula" : "min(sweeps(cursors(A,B),channel(AD),[1...inf]))",
    "cursors" : [{A : "100ms"}, {B : "200ms"}],
    "sweeps" " [ 1, 2, 3],
    "experiment" : [ "filea.nwb", "fileb.nwb"]
  ,
    ...
  ]

Separate issue https://github.com/AllenInstitute/MIES/issues/273.

• [x] FindLevel

  how to pass the direction? how to pass failing/rising? findlevel(RMS(sweeps(cursors(A,B),channel(AD),3;4;5;6;7;8;9;10)), sweeps(cursors(C,D),channel(AD),3;4;5;6;7;8;9;10))

Separate issue https://github.com/AllenInstitute/MIES/issues/281.

[ukos-git]

I kindly ask you to read the documentation that I added. https://github.com/AllenInstitute/MIES/pull/245/commits/61fd68aef6094b8e70e26567e45aa0b4eb2f671a

I will polish the GUI a bit more and do some fixups. I want to bring the remaining bullet points to separate issues to discuss the problems that are only solvable with more time on the associated PR. We should first quit these remaining discussions before proceeding further and as there are 100+ issues open in MIES, 5 more should not be a problem.

I strongly recommend to finish the associated PR first. It is mainly data alignment problems that remain to get tackeled. e.g. derivative*data is not a problem but but normalizations using data/max(data) is currently undefined so we need some formula combinations from you @timjarsky and what should be the defined calculation result from it. Best with simple arrays using range(0,100),range(0,200,2) examples or so.

[ukos-git]

A few thing remain to be done in the MR:

• [ ] preserve wave scaling for all operations
• [x] define the result of some concatenated operations to ensure that we use the right data alignment for these operations.

Add Curosr hooks

do we still need that?

One trace per channel number (i.e. if you have AD0 and AD9 shown in the data browser, we want to have two traces)

what was the problem with that again? On my end it was working since forever but maybe I need to test with the DataBrowser to see that.

FindLevel how to pass the direction?

Should I add the findlevel? - I really like `findpeak? more

FindPeak [/B=box /I/M=minAmp /N/P/Q/R=(startX,endX )] waveName The FindPeak operation searches for a minimum or maximum by analyzing the smoothed first and second derivatives of the named wave. Information about the peak position, amplitude, and width are returned in the output variables.

[t-b]

min/max has already the correct alignment of data. Mean got it wrong. I'll add some parametric tests to test it and fix it then.