sync_stream_manager crashed for 30+ videos

[wanghlsara]

Very awesome work, this helped me a lot.

I used videos from more than 30 cameras to calibrate external parameters. Each video has about 15,000 frames. I set fps_sync_stream_processing = 2 and save_tracked_points_video = false, but it is still very slow when executing "calibrate_capture_volume", and the program crashes in the middle of "sync_stream_manager.process_streams" after running 2 hours.

Is there any simple way to modify the code for me? For example, do not display the progress dialog box "Landmark Tracking and Triangulation", or ignore the processing of video time synchronization (assuming that the videos are all recorded simultaneously), directly detect the chauco maker of each video and then merge it into xy_charuco.csv?

Looking forward to your answers.

[wanghlsara]

I've made the progress dialog not display by commenting out the code that calls SynchedFramesDisplay. I observed the Windows Task Manager. When the program crashed, the sum of committed physical memory and virtual memory reached the limit. The cause of the crash should be that the memory was exhausted.

[mprib]

Well, first off, let me just say 'thank you' for stress testing this thing.

30+ cameras x 15,000 frames is definitely a lot to process, but I want this tool to be robust. So let's talk...

While there are some little hacks that might improve things, what you describe here definitely feels like the robust way to move forward:

...directly detect the chauco maker of each video and then merge it into xy_charuco.csv

As you note, this requires that all frames be truly synced at recording so that the frame_index represents the same instant in time across cameras. Reconfiguring the processing pipeline to work like that is not a special challenge, but I also wouldn't describe it as simple. While it might only take a day of focus to get that done, it might be another week before I can carve out that day of focus.

When I have the bandwidth to look at this, I will touch base on this issue again to confirm with you that you are still looking for a solution (or at least willing to test the solution I come up with).

Regards,

Mac

---

Addendum:

Some quick ideas and follow up questions to clarify the situation

Modify charuco tracker

https://github.com/mprib/caliscope/blob/2119901e67e28bde60b73859f689331be9037ff5/caliscope/trackers/charuco_tracker.py#L43C1-L46C1

In charuco_tracker.py comment out these lines and it may reduce the overall system demands:

        if not ids.any():
            gray = cv2.flip(gray, 1)
            ids, img_loc = self.find_corners_single_frame(gray, mirror=True)

Reduce number of frames

I will say that 15,000 frames is a heck of a lot and I sincerely doubt that this is really providing all that much value.

What frame rate is being used here?

My guess is that you could get comparable results and dramatically reduce processing time by sampling only once each second . You want the sample of calibration boards to be distributed broadly across the capture volume. Having thousands of data points that are immediately adjacent to each other is not going to really provide much additional value.

[wanghlsara]

Glad to see your reply. Following your suggestion, I set fps_sync_stream_processing = 1 and removed the code for detecting mirror images you mentioned. After 3 hours, the xy_CHARUCO.csv file was generated. As for the previous idea of directly detecting the chauco maker of each video and then merge it into xy_charuco.csv , I will try to write a simple but useful code later just for testing. So, I think the crash problem is no longer a problem for me, thank you again.

I will say that 15,000 frames is a heck of a lot and I sincerely doubt that this is really providing all that much value. What frame rate is being used here?

The frame rate being used is 50 fps, and I am going to re-record the videos using 30 fps in order to reduce number of frames. I conducted this experiment in a narrow and long rectangular area, It takes about 4-5 minutes for me to move the calibration board from one side of the area to the other, this may resulted in a large number of frames. You mentioned that too many points won't really provide much additional value, So I want to try to use fewer points in the code. I would be very grateful if you have any suggestions for me to modify the code.

[mprib]

I made similar comments on the other issue, but if at all possible you could use a "mirror board" with the mirror image of the charuco on the back. If you already have the intrinsics calibrated from a very flat board, then it's not terribly imperative that this mirror board be perfectly flat (being pressed between a couple sheets of plexiglass may not be necessary). As long as the board doesn't warp too much you could still get a decent enough initial estimate from the stereocalibration and for the bundle adjustment to proceed smoothly. For the bundle adjustment, it doesn't really matter about it being flat at all, just that the same points in space are identified across frames.

Having cameras opposite each other able to link together by viewing the same point in space from either side of the sheet of paper may be helpful. And to reiterate: even 30 fps may be more than is needed for this process. You could start with 5fps and see how it goes.