update pen data series and run boundary detection logic

[isolver]

Change things to rely on the sample status / state field.

• FIRST_ENTER = sample series start boundary
• FIRST_PRESSED = sample run start boundary

Notes:

• See some of the comments in issue #112, which include some of the discussion trail that has led to this issue being openned.
• Need to add extra logic to handle status of first and last sample of file correctly
• For hdf5 files with trial segmentation, need to handle status of first and last sample of each trial correctly

[isolver]

@Mark-Torrance, I think the following maybe what you suggested at the very start of all this, but now I see the light....maybe.....

If we are not going to support the idea of parsing txyp and xml file samples into Series based on an ISI threshold, then I think we should:

• create 1 default series which contains all the samples from the data file.
• disable all the series related actions for projects created from these file types.

Otherwise it is just confusing for people, even more so if we do enable some form of sample Series definition for these file types in the future.

[Mark-Torrance]

If we are not going to support the idea of parsing txyp and xml file samples into Series based on an ISI threshold, then I think we should:

• create 1 default series which contains all the samples from the data file.
• disable all the series related actions for projects created from these file types.

Otherwise it is just confusing for people, even more so if we do enable some form of sample Series definition for these file types in the future.

That sounds good. Don't do this on the basis of filename, though. Have a setting for "only process press data". We tell users that they must choose this if their data is from E&P / HandSpy (current versions of...), but also they may use it if they want to ignore hover data in GetWrite generated data files. I can imagine wanting to do that for some purposes.

[isolver]

I have implemented these changes. Same logic is used for all import file types:

• Series are parsed using sample status field state FIRST_ENTER for all file types
• Runs are parsed using sample status field state FIRST_PRESS for all file types. If a Series contains no FIRST_PRESS status samples, the pressure value of samples in the Series is used to try and detect Runs (i.e. Run detection for the Series falls back to old logic if for some reason no FIRST_PRESS samples are found. Never seen this actually happen, but why not.).
• ISI is not used at all anymore. It is not even calculated or stored anymore, speeding up file loading a bit.
• Have not implemented your suggestion for "Have a setting for 'only process press data'.". For current file types, is it necessary given there are already settings for:
  • doing stroke detection on Runs only.
  • trimming hover samples from start / end of Segment being created.

Please let me know on this one.

Still TODO related to new Series / Run parsing:

- [ ] for hdf5 files that have trial segments, ensure each Series does not span > 1 trial. A Series can not start in trial 1 and end in trial 2 from markwrites POV, although in theory it is possible in an hdf5 file depending on how the experiment is designed and what values are used to do trial selection.

[isolver]

@Mark-Torrance : Am I actually correct about the TODO stated in the above comment? i.e. should pen series / run/ or stroke parsing be affected if l1 segments are created for trial periods?

I'm now thinking that it should not so I should not actually do the above todo. In other words, it is fine if a series / run/ or stroke starts in 1 trial and ends in the next. That would just be representing the real situation, instead of artificially changing the start / end time of any such unit that crosses a trial segment start or end time.

[Mark-Torrance]

Good point. Thinking about it I think you're absolutely right. The same applies to runs. If we want participants to start the trial with the pen out of range, or not pressed, then we need to build that into the design. The experiment should tell us how things actually are.

What we don't want is a situation where the selection can be expanded beyond the current trial (even if the trial start is mid-series or mid-run, but that's a separate issue.

[isolver]

On Tue, Jan 26, 2016 at 9:45 AM, Mark Torrance notifications@github.com wrote:

What we don't want is a situation where the selection can be expanded beyond the current trial (even if the trial start is mid-series or mid-run, but that's a separate issue.

Agreed.

[isolver]

So closing this, as it is fully implemented.

[Mark-Torrance]

This is just a point of information, but good to document here:

When GetWrite adds status to the hdf5 what ISI criterion is used to identify series onsets (FIRST_ENTER)? Could you point to the place in the code that does this so that it can be adjusted if needed?

[isolver]

Markwrite makes no changes to hdf5 file pen sample status/state values; they are populated at runtime and saved to the hdf5 file.

FIRST_ENTER sample status state is set by iohub based on receiving a wintab WT_PROXIMITY msg. WT_PROXIMITY msg's occur when the pen first enters the detectable surface / area and when the first pen leaves the detectable surface / area. However WT_PROXIMITY events contain no position / pressure / etc. data.

When a wintab WT_PROXIMITY msg is received, and the msg's lParam & 0xffff is > 0, this means the pen just entered the detectable region. iohub effectively uses this to set a flag that is consumed by the next pen sample event (wintab WT_PACKET msg) is received from the device, adding FIRST_ENTER to the state of the iohub pen sample being created.

So ISI information is no longer used at all in markwrite, it is not even being computed anymore.

[Mark-Torrance]

Excellent. Thanks Sol. Moving that to the wiki.