July 2021 Notes: Superframing and More Image Reassembly

[john-judge]

7/1/2021

Continue from the achievement at the end of #18

TO DO:

• [x] Ensure that RLI works as expected now that 475-frame is reinstated.
• [x] Test deinterleave on other settings -> Result: seems to work well, it's just that without superframing it seems we're missing some frames.
• [x] Test remapQuadrantsOneImage within PhotoZ
• [x] Visualization. Integrate Matplotlib C++ or other plotting lib that's more powerful than FLTK?) -> so far, installed OpenCV into PhotoZ, and researched compatibility with FLTK.

7/5/2021

• Image 350/450 do not deinterleave properly -- suspect remapQuadrants is incorrect

• PhotoZ does not collect bright data unless TurboSM has been used just before. Something missing from PhotoZ init.

Steps taken to produce the following image: 1) Disable call to remapQuadrantsOneImage 2) Ran TurboSM acquire one image, closed TurboSM 3) Acquired 475 RLI points 4) Plotted 350th image (light on), with Python quadrant-remapping enabled

Improved overall efficiency of image reassembly by reducing number of tmpBuf allocations and copy-overs. Noticeably faster -- by an estimated order of magnitude. Correctness seems to have been preserved.

Now able to remap and flip quadrants entirely within PhotoZ!

2048x50 Readout

Doesn't seem to be successful every time, variation in the raw data. This could suggest the camera is still trying to read out a large 2048x1024 image (instead of downsampling/reducing resolution) and we're only getting pieces of it here.

TurboSM:

Or maybe the PDV channel threads are too slow to keep up with the camera at this frequency? Need superframe? After all, we did just jump from 200 Hz to 2000 Hz

1024x160 Readout

Back down to 1000 Hz. This one seems to follow a different deinterleave pattern (alternating-row stripes appear when zoomed in to pixels). However, maybe it is less likely to suffer from missing frame issues.

Superframing

Attempting to use

pdv_setsize(pdv_pt[ipdv], width(), height() * get_superframe_factor()

to introduce superframing. It seems that dimension validation doesn't match up -- pdv_get_imagesize is reporting a superframe factor of 2 when I asked for a factor of 10. Also, pdv_setsize dimension changes seem to persist after the first call, disallowing further modification -- more investigation needed.

Superframe seems to be possibly working (at least passing validation). However, now the LED stays on for ~10 seconds. I reformatted the raw image organization scheme. Now, all data from a single channel is contiguous in one block, followed by a block for the next channel, and so on. Need to have Camera image reassembly functions adopt this convention, but this is a minor change. Then I can see if superframing is working (I'm not optimistic).

2:40 PM

With the introduction of pdv_setsize, it seems opening TurboSM before PhotoZ right when the cam is booted is no longer needed. Also, having restarted the camera, it no longer takes ~10 seconds with LED on -- now seems to be back to how it worked before. However, I'm not seeing the data read out properly to file. Attach debugger maybe? Could it be that the camera is just reading back all zeros like I have seen before?

3:08 PM

Fixed some memory arithmetic bugs. First, not getting back meaningful data (a few odd columns) Then, opened TurboSM. Suddenly, we have plenty of bright data in PhotoZ, but seemingly not in the correct dimensions. The cam height is stuck at 512 (left there by TurboSM) and the image height is 500 (height * superframe) Then try PhotoZ RLI again. Returns back to first state.

How to superframe? Maybe edit .cfg files? Before that heavy-handed of an approach, will browse TurboSM code some more.

4:14 PM

Attempted superframing (x10) by creating a copy of a .cfg file called 'DM2K_2048x500.cfg' However, I'm not sure this worked properly -- does the camera know that PDV is going to treat the data as coming in periods of 50 rows (for a total of 500 rows per image)? Alternatively -- suspect problem with pointer arithmetic?

4:53 PM

There was one pointer arithmetic problem, but still not correct Looked up .cfg directives and tried setting frame_height = 50 -- but this directive may be deprecated according to some PDV docs. Same question -- looks as if a 2048x500 image is being read out instead of 10 2048x50 images? As a reasonability check, read out 2048x1024 at this point -- appears to be something gone wrong in the transition to the new channel-contiguous blocks of memory. Let's fix that before proceeding.

[john-judge]

7/6/21

From TurboSM:

pdv_start_images(pdv_pt[thread_id], loops + (factor > 1 ? 1 : 0));

where second argument is count -- appears that this needs to be the number of superframed images? Testing modification to Camera::start_images():

    // count    number of images to start. A value of 0 starts freerun. To stop freerun, call pdv_start_images again with a count of 1.
    //pdv_start_images(pdv_pt[ipdv], 0);
    pdv_start_images(pdv_pt[ipdv], get_superframe_factor()); // changed COUNT arg from 0 to superframe factor.

Use this together with pdv_setsize and the height to a factor of superframe? Or with frame_height cfg directive?

Also, previously:

As a reasonability check, read out 2048x1024 at this point -- appears to be something gone wrong in the transition to the new channel-contiguous blocks of memory. Let's fix that before proceeding.

Still seeing the trouble where TurboSM must be run before PhotoZ starts acquiring properly.

10:57 AM

Still hunting down memory pointer arithmetic bug.

12:00 PM

Bug resolved! Turns out walking pointers are much more elegant and easier to debug than arithmetic assignments to pointers.

Testing with pdv_setsize and start_images with count arg passed, and with the original .cfg file. First a regression check with 200 Hz setting. This setting has a superframe factor of 1 and still starts images in freerun mode (count = 0). Result: Need to disable pdv_setsize call for this setting, I think.

[john-judge]

7/7/21

Disabled pdv_setsize and reverted count arg call for the 200 Hz setting, and as expected the previous correct behavior remains (no superframing at this setting). Since is the case that TurboSM must be run first before images in PhotoZ are correctly read out. Maybe TurboSM is setting the camera gain or something?

For now, however, focus on superframing investigation with 2048x50 at 2000 Hz setting.

Superframing Investigation: 2048x50 cfg (2000 Hz)

• Testing with pdv_setsize and start_images with count arg passed as the superframe factor, and with the original .cfg file. RLI is slow, LED is on for ~10 s, and the data I am seeing is essentially blank. This error can't be in processing, because at the very least I should see some data in Frame 0 in the raw data (which is printed to file before processing is started).
• Started TurboSM and switched it to 2048x50 cfg (2000 Hz) setting. Back to PhotoZ, result looks somewhat reasonable (if the frame were supposed to be dark):
• From this, it may help to look at images that are not the beginning of a superframe. Frame 355.
• Seems need to run TurboSM at 2000 Hz after every single PhotoZ acquisition at 2000 Hz, or no data is read out at all.
• Idea: revert to 2048x500 .cfg file? Maybe later. Clearly TurboSM doesn't use this route. More TurboSM code investigation now.

Questions based on TurboSM investigation

• What does the @REP serial command do? Is it necessary for superframing? From sm.cpp:

  sprintf(command,"@REP %ld",repetitions);
  SM_serial_command(command);

• Seems like pdv_serial_command and pdv_serial_write are interchangeable (documentation). But currently we're using pdv_serial_write and we're not appending the serial terminator \r\n -- is this a problem? I switched over and did not notice any improvement.
• What do all the 3-character @ commands for the 2K camera mean? @RCL, @SPI, @PSR, @TXC, @SEQ, @PRG, @AAM mean? How might one query the camera for its current state?
• To superframe, should I start images (pdv_start_images) in freerun mode or request the number of images that make up one superframe?
• How is \SMSYSDAT\image_chan_order.txt used? We don't have this file, but I determined the camera channel order through trial and error)
• How is \SMSYSDAT\draw_seqROIs.txt used? We don't have this file either. Are some pixels cropped out of the final image for the settings 512x160, 512x80, 256x60, and 256x40 ? Based on what I know, it seems we will read out 2x or 4x more pixels than the final image for these settings.
• I call pdv_setsize with a superframed height immediately after the initcam/cfg file calls, and before any further serial commands. Is this correct timing or should this be called later?

To-do next

• [ ] Mirror Chun's main.cpp init sequence more exactly and test. not providing results, based on below
• [ ] Discuss above questions with Meyer so he can send an email to Chun.

1:04 PM

Attempted adding this just after initcam script calls:

    sprintf(command, "@RCL %d", m_program);
    serial_command(command);

Test: RLI at 200 Hz w/ above, no change (still needed TurboSM to init properly)

200 Hz did not work until I reverted serial_command to serial_write and removed @RCL write

[john-judge]

7/8/21

Changed raw size alloc in DataArray.cpp to match the expected display width and display height (since cam.width and cam.height are the cfg quadrant dimensions).

Testing different initcam instruction orderings. Open channel after script load?

Moved channel-opening from MainController::takeRli into the Camera.init_cam just after the cfg files are loaded. Seems no difference -- still requires TurboSM to run beforehand.

Now note where else is Camera.init_cam called -- how might this affect / create a difference from Chun's recommended init seq?

Tried adding @RCL back. According to Little Joe book, RCL # is recall settings -- accesses settings from the camera's storage. (Little Dave seem to use the same commands? Chun's code sends these commands to 2K's, at least). Result: image was read out properly. Does it still require TurboSM to start first? Yes.

In general, the result when reading out before TurboSM has been either no data or just a small amount of data -- apparently just the very top rows and very bottom rows, not in the expected location either:

11:30 AM

Replaced pdv_wait_image with pdv_wait_image_raw. Result: no difference. Reverted the pdv_setsize change for 200 Hz. After the first readout with pdv_setsize, all subsequent readouts give no data.

From Little Joe book on @REP:

The last segment in any sequence is the accumulation (or integration) segment. This may be a short do-nothing operation, or may involve dithering on long exposures to reduce dark current.

Repetitions -- Adjust the number of times the accumulation segment of a program is repeated.

So apparently the @REP serial command has nothing to do with dimensions or superframing, and even if it's as relevant to Little Dave as it is to Little Joe, it's not the missing piece.

Testing adding "@SEQ 0" just after the RCL # serial command. Result: camera is slow, taking images takes ~ minutes.

Removed the second RCL call (x4) from cam.program

Maybe the camera hung indefinitely because there was no matching SEQ 1 to start the camera? added these commands at the end of cam.init_cam, mirroring TurboSM:

    sprintf(command, "@TXC 0");
    serial_write(command);
    sprintf(command, "@SEQ 1"); // start 
    serial_write(command);

but the result is unchanged: camera is slow, ~ min. Reverted.

SM_take_tb Superframed Acqui Example

pdv_start_images(pdv_pt[thread_id], 4);
image_ptr = pdv_wait_image_timed_raw(pdv_pt[thread_id], (unsigned int *)&times[thread_id][0], 1);
pdv_start_image(pdv_pt[thread_id]);
for (int i = 0; i < factor; ++i) {
    memcpy(data_ptr[thread_id], image_ptr, frame_length / factor);
    image_ptr += frame_length / factor;
    data_ptr[thread_id] += image_length / factor;
}

Attempted to mirror PDV doc to read images out in multiple buffers, increasing multibuf from 4 to superframe factor -- Result: long-duration acquisition for program 6, still requires TurboSM to be run first, and only works for the 1st frame of each superframe, but the image seems correct: Repeated it a second time -- like always whenever pdv_setsize is used, the second time reads out no data.

Attempt Same But 4-Multibuf, Superframing still

2000 Hz -- LED was on ~9 s (improvement!) But only the first 1/10 (superframe=10) of each superframe was filled (only frame 1 of 10). Reattempted with having the loop fixed to superframe=1. Took several times longer to acquire. Result: similar. Seems only first of the 4 multibufs has any data.

[john-judge]

7/9/21

Meyer's email from Chun:

For NeuroCCD or any camera single-edt/pdv-channel

pdv_flush_fifo(pdv_pt);
pdv_multibuf(pdv_pt, 4);
//has to be 4 for 64bit. (You are missing this line)

pdv_start_images(pdv_pt, loops);
for (int j = 0; j < loops; j++) {
image_ptr = pdv_wait_image_raw(pdv_pt);
memcpy(data_ptr, image_ptr, frame_length);
data_ptr[0] += image_length;
}

The loops can just be the number of frames for a start.

To reduce interrupts and ensure not to lose any frames (especially at high frame rate), we make super frames, meaning each frame will have height of orig_height*factor, then the loops = num_frames/factor.

For 32bit, we used to read in another config that sets the height 10x (80x800 from 80x80). For 64bit, we do

pdv_setsize(pdv_pt, config_num_col, config_num_row*factor);

For DaVinci, list of serial commands: DV-2K-CamInit.txt

[john-judge]

7/14/21

Summary of progress from past couple days:

• pdv_setsize now seems to be working now that I moved the location of its call to after the PSR/SPI serial commands. For prog 1 (2000 Hz) I varied superframe factor at values 1, 2, and 10, and the frames appeared invariant to these changes, suggesting that 1) memory location arithmetic is all correct (w/o superframing had already been verified with other tests) and 2) pdv_setsize and the final image dimension are working together successfully
• Also, with that change, the subsequent acquisition runs aren't blank when pdv_setsize is called.
• The following observation applies to both prog 0 (200 Hz) and prog 1 (2000 Hz): Still requires TurboSM to be run beforehand to read out a proper image. If TurboSM is not run beforehand, it appears only a piece of the image is visible -- for 200 Hz, this looks like the top and bottom rows without the middle rows, and the field of view is larger (dark areas).
• Chun suggested maybe PSR, SPI problems. I tried permuting to pdv_serial_command and adding/taking out RCL, SEQ, TXC, and adding/removing trailing \r from the commands. pdv_serial_read did not seem to get any responses back from the camera for PRG or RCL queries. Removing some of serial commands prevented any image readout, so it seems to confirm that the camera is getting the command in some form.
• In light of the above point, I suggested to Chun that maybe we are missing the REP command that TurboSM appears to use in SM_initCamera > setFocusRep in sm.cpp. Asked for Chun's thoughts.
• Overall pretty pleased that superframing is essentially working and that acquisition and reassembly are fairly efficient.

[john-judge]

7/20/21

Adjusting start line lib by trial-and-error.

2048x100

Final start_line_lib value is 888+142 (last image)

1024x320

1024x160

CDS subtraction width seems to be half of cfg width, not always fixed at 1024.

Correct start_line_lib:

576 1038 766 926 766 926 846 926 - start_line_lib

[john-judge]

7/20/21

hbin and ccd_lib_bin fix did the trick!

Pulled the correct values from sm_config.dat. Also corrected the start_line_lib as well.

The following were read out one after another, without having run TurboSM beforehand, and without any camera restarts.

1024x320

1024x160

2048x100

2048x1024

512x160

512x80

256x60

256x40

[john-judge]

Work on this project now continues in https://github.com/john-judge/PhotoLib