Include custom tiny analysis steps into dragonviewer

[dneise]

I spent an entire day on this (and learned a little PyQt4 in the process) ... not sure if it is really of any use ... but in the morning I liked the idea :-D

The story goes like this:

In fact-tools we have the nice event viewer, which can produce a lot of plots and stuff, based on what has been added to the event. While looking at events, and especially at the sampled time series, I often want to plot, print, or somehow highlight a certain quantity, like:

• show the mean over the time line, maybe as a horizontal dashed line?
• show me, where the arrival time would be ... I cannot see that by eye.
• print something ... maybe even a statistic about the events, I've seen so far.

---

Now adding features in Python is easy. But PyQT4 I personally find difficult to use, and merging Qt4 and matplotlib, can feel even more difficult for people. So I though it would be nice to be able to hook simple functions, into the dragon viewer, which get the event, and print or plot something we like.

One might use the dragonviewer like this normally:

    cd to/my/data/folder
    dragonviewer some_data_file.dat

Now assume, one looked at some events and suddenly wanted to calculate something on a certain time line and print it out. One knows how to code it ... maybe somehow like this ....

def my_funny_func(channel, gain, data, stop_cell):
    print("awesome result of channel {channel}: {result}".format(
        channel=channel,
        result=data.mean() + 3*data.std() - stop_cell)
    )

... but one doesn't know how to include this into the viewer ... and it seems too cumbersome to create a complete app for this.

---

Now all the user needs to do, is this:

# dragon_analysis.py
def my_funny_func(channel, gain, data, stop_cell, axis):
    print("awesome result of channel {channel}: {result}".format(
        channel=channel,
        result=data.mean() + 3*data.std() - stop_cell)
    )

So implement the function one has in mind in a file called dragon_analysis.py in the current folder. And start the dragon viewer again.

The viewer has now, on its upper right side, a text box, with the names of all functions, which have the correct signature, i.e. the accept parameters with the names {'axis', 'stop_cell', 'data', 'gain', 'channel'}

On the lower right side, the stdout of the functions, the user has implement gets shown.

Via the axis parameter, one can even manipulate the plot. In case there is no dragon_analysis.py in the python path, an example_analysis.py from the dragonboard module is imported. This is how this looks at the moment.

[maxnoe]

I like the possibilities! Did not have time to glance over the implementation or test it out. What I would like would be a command line option for the analysis script.

Like a script that actually can run on it self, but when run with the dragonviewer its results are shown in the viewer.

Might be hard to implement.

Maybe this could help us: http://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path

[dneise]

I agree, specifying the path to the "analysis.py" is much more clear for the user, than having to remember the name "dragon_analyis.py". Thanks for the SO link, this indeed should be a way to implement this.

The second part, letting the analysis be executable on its own and be pluggable into the viewer, seems indeed a little harder, but I still find it interesting. One thing to keep in mind is maybe, that this is exactly what I thought, a user who is currently playing with the viewer, mind not want to do: "Write a real executable analysis", because it poses a little overhead on what he might actually want to achieve.

Anyway, I think by defining something like a processor interface for a possible analysis functions, which might look like this:

def analysis_function_candidate(analysis_event, channels=None, axis=None):
    """
        parameters:
          * analysis_event : {"event": dragonboard.io.Event, ...}
          * channels: [(pixel, gain), (0, 'high'), (3, 'low'), ...] or None
          * axis: matplotlib.axes._subplots.AxesSubplot
        possibly calculate something on analysis_event
        possibly plot into axis
        possibly print something
        possibly attach result into analysis_event
    """
       # do the stuff
    return analysis_event

We could maybe get to the point, where user written analysis scripts, can also be imported into the viewer. The viewer imports the script as a module, scans for functions, which have the right interface, and we are done. But this would need users to implement their analysis scripts in a way, which makes use of functions that have the processor interface. For example like this:

def func1(analysis_event):
    foo = analysis_event["event"][0]["high"].mean()
    print(foo) 
    analysis_event["foo"] = foo
    return analysis_event

def func2(analysis_event, axis):
    bar = analysis_event["event"][5]["low"].std()
    axis.axhline(bar)
    analysis_event["bar"] = bar
    return analysis_event

all_analysis_functions = [func1, func2]

if __name__ == "__main__":
    args = docopt(__doc__)
    event_generator = dragonboard.io.EventGenerator(args["--inputfilepath"])
    calib = dragonboard.calibration.TimelapseCallibration(args["--calibfilepath"])

    for event in tqdm(event_generator):
        analysis_event = {"event": calib(event) }
        for analysis_func in all_analysis_functions:
            analysis_event = analysis_func(analysis_event)     

If by chance or by convention a certain fraction of analysis scripts would look similar to this, we could maybe achieve what you proposed ... but ... as one can see for func2, one without the viewer, we need to provide dummy Axis instances into the analysis functions, so the can plot into something. This feels pretty unnatural in a normal analysis script...

I need to further think about this .... so many possibilities. Thanks for your feedback, I certainly want to get your first proposal running.