Collecting images on camera 2 causes break with newer version of Imagine

[Callahanra]

I was recording on Camera 2 in a single plane for 2000 images at a exposure of 0.03 ms and Imagine got through ~1400 images and then it triggered a breakpoint. It won't allow me to continue through the breakpoint and I had to break out and stop the debugger mode . It happened 2 times with the exact same settings. The JSON and Imagine files appear to be saved if you would like those.

This is the section where the break was highlighted in the Imagine code: void FastOfstream::write(const char* moredata, int morecount) { double timerValue = timer.read();

//pad/copy (for alignment, otherwise sometimes we can use moredata directly) then write cur buf (repeatly)
//leave the remainder in cur buf
while (true) {
    if (this->datasize == bufsize) {
        OutputDebugStringW(L"Flushing output file stream from fast ofstream write\n");
        flush(); //NOTE: will update datasize too
        if (!isGood) break;
    }

    int amount2cp = min(bufsize - datasize, morecount);

    /* this has 10x performance hit when in debug mode
    for(int i=0; i<amount2cp; ++i){
    buf[datasize++]=*moredata++;
    }
    */
    //alternative:
    //takes about 1.9 milliseconds for a full frame on PCO.Edge 4.2
    **memcpy_g(buf + datasize, moredata, amount2cp);**
    this->datasize += amount2cp;
    moredata += amount2cp;

    morecount -= amount2cp;
    if (!morecount) break;
}//while,

return;

}//write(),

[Callahanra]

I thought I had bolded the line that was noted in the code at the breakpoint, but it doesn't look like it worked. Its this line: memcpy_g(buf + datasize, moredata, amount2cp)

[Cody-G]

2000 images at a exposure of 0.03ms

Maybe you mean .03s? .03ms is too short to be useful in most cases.

Definitely keep those files, they'll be very useful for debugging. I can't say a lot without more info / debugging it in person, but the bug is probably not in this function. Rather this function is getting called with invalid arguments. It's possible that the data buffer was never allocated or it's getting sized incorrectly.

[Cody-G]

...And the fact that it happens after 2000 images may not be a coincidence. <--Nevermind, I misread. I see that it's 1400. I think that's about the size of the memory buffer for camera data (we don't need a buffer big enough for all the images because the memory buffer is constantly being emptied while we write images to disk).

[kdw503]

I just checked this issue with Rebecca. But we couldn't regenerate this error yet. We need to keep looking this problem.

[Callahanra]

So this has been happening again, at least it's breaking on the same line. This time only camera 1 is on. I've messed around a bit with number of frames and exposure times, with nothing popping out as specifically causing it. I do always have the camera image cropped down to an ROI. Doesn't look like I can upload the .ai or json files here, but I can send them to you both if you would like.

As a separate, but related, request, is it possible to have Imagine close the laser shutter when a breakpoint is triggered? I am just worried about the extra bleaching while I am closing the program and restarting it (and the shutter and laser stay open during that time.)

Oh and yeah I def meant 0.03 s ;) (though now I've tried 0.05 and 0.02 and on whatever unlucky recording the breakpoint has been triggered)

[kdw503]

Sorry for happening this again. We will keep looking into this problem. If you send me your json and config(if you save your GUI parameters into this file.) files, it would be helpful to regenerate this error.

About closing laser shutter when the program breaks, that is good idea. I will check if this is possible. Until then, how about turning off the laser by hand? Or, how about we provide separate small program which just turning off the laser shutter?

[Cody-G]

How would it help to make a separate program for that? We would still need to communicate with Imagine to know when there is a crash right?

On Mon, Oct 30, 2017 at 11:04 AM, Dae Woo Kim notifications@github.com wrote:

Sorry for happening this again. We will keep looking into this problem. If you send me your json and config(if you save your GUI parameters into this file.) files, it would be helpful to regenerate this error.

About closing laser shutter when the program breaks, that is good idea. I will check if this is possible. Until then, how about turning off the laser by hand? Or, how about we provide separate small program which just turning off the laser shutter?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/HolyLab/Imagine/issues/60#issuecomment-340493230, or mute the thread https://github.com/notifications/unsubscribe-auth/AE78hGpuToos4sU1fTMtVfDpV9tI4Xnxks5sxfN0gaJpZM4QIBG7 .

[kdw503]

I mean user executes the program(this program only turns off shutter and terminates) when a crash happen. But this program can be a base code to implement what Rebecca wants later. My idea is,

1. When imagine start, imagine runs this separate program (we can call this as watchdog).
2. Watchdog program set a variable in the shared memory(with Imagine) as zero .
3. Imagine regularly set the variable as one.
4. Watchdog regularly reads the variable if it is one and reset it as zero.
5. Watchdog closes the laser shutter and terminates if the variable is not changed as one in some amount of time.

[Cody-G]

I see I think that makes sense. From the user's point of view it's still one program right? I think we'll want to use a lock or an atomic bool to make sure there's no problem if the two threads try to modify the shared variable at the same time.

[kdw503]

If user should execute the program(Laser turn off), it is a separate program. But, if we implemented the watchdog program, user would not recognize it as a separate program. And about the shared memory, it looks we don't need to worry about the situation because windows provides 'Named Shared Memory' feature for communicating between processes. I'm not sure though..

[Cody-G]

I see, I'm not familiar with that Windows feature. Another thing to keep in mind is that the watchdog should probably be notified of a failure in any one of Imagine's threads. So if the ai thread crashes but the camera worker thread does not, then the watchdog should still care. I guess each Imagine thread could have a different variable for checking in, but it seems to me like there should be a more standard way to accomplish this.

On Tue, Oct 31, 2017 at 12:09 PM, Dae Woo Kim notifications@github.com wrote:

If user should execute the program(Laser turn off), it is a separate program. But, if we implemented the watchdog program user will not recognize it as a separate program. And about the shared memory, it looks we don't need to worry about the situation because windows provide 'Named Shared Memory' feature for communicating between processes. I'm not sure though..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/HolyLab/Imagine/issues/60#issuecomment-340813310, or mute the thread https://github.com/notifications/unsubscribe-auth/AE78hDOsxnTj3Y4fTZ0gsbWdaiVSgSooks5sx0YrgaJpZM4QIBG7 .

[kdw503]

Oh, I didn't know that even if one of thread crashes others are still running. But I think in our case, the thread which includes the control of daq is the most important one. If the thread is still alive, we can close a laser shutter after recording(if we care a crash happening only in recording status). And as you said, I'll study more if there is a more standard way.

[kdw503]

I just tested watchdog. It works well until now. It seems all thread stop if a certain thread crashes. I think that's because they are all belong to the same process.