Dastard crash when data dropped and trigger rate is high

[joefowler]

Error when we have a very high photon rate at BESSY-II. Seen twice:

DATA DROP. firstFrameIndex 0, droppedFrames 248
panic: runtime error: slice bounds out of range [-1:]

goroutine 28700547 [running]:
...

The error trace says error is in triggering.go:48, which reads

copy(data, stream.rawData[i-NPresamples:i+NSamples-NPresamples])

Clearly we are trying to go back before the existing data, and apparently by exactly 1 sample. Sounds like an off-by-1 error somewhere?? See if we can debug on a live machine, but also try to sniff it out.

[joefowler]

Okay, we determined that this happens only when changing record lengths, as soon as the new length takes effect, whether longer or shorter than before. Weirdly, it always seems to have i-NPresamples equal to -1.

This is useful clues towards finding the off-by-one error. It's not in the process of padding or triggering exactly, but in the process of changing record lengths. Perhaps I mis-aimed the "don't trigger before here" pointer by one?

[joefowler]

I suspect the problem is only in EMT=Edge Multi-Triggers, though I can't prove that yet. I have made a work-around. (See branch drop_data_offby1, all commits on or before Oct 26.) It consists of:

• Before copying raw data into a triggered record, test that the copy won't try to start illegally. Here, illegally means trying to copy data from the slice starting at a negative index.
• If the copy would be illegal, write a terminal message and return a nil record.
• Modify the triggering algorithm so that after all triggered records are created, look for and remove any nil values before passing the slice on to the analysis and publishing parts of the program.

Still...it would be best to find where this is happening in the first place, and fix it.