OpenCL Buffer Error

[NicerNewerCar]

This error has been reported by Cesar and Dajung

Destination buffer does not have enough room!

https://github.com/BrownBiomechanics/Autoscoper/blob/794539cf53e79114091cc7cf3d7b08e1a78bd291/libautoscoper/src/gpu/opencl/OpenCL.cpp#L992-L999

[NicerNewerCar]

After a quick initial investigation, it would appear that the data loads in fine. The error happens when the NCC is calculated for the first time.

[NicerNewerCar]

Disabling the NCC calculation yields the following result:

While this loads in, I would not consider this "fine" as I mentioned above.

Inspecting this volume in Slicer, I cannot find anything out of the ordinary compared to other data (from the same dataset) that works fine.

My plan moving forward is to do a complete comparison of the rendering pipelines for the broken and working data.

Update

Playing around with the X-ray intensity the volume actually looks fine. Maybe this has something to do with the error?

[NicerNewerCar]

The error is caused when the projected image resolution is higher than the specified render resolution. For the above model that is 764x467 vs 512x512. Simply increasing the specified render resolution won't suffice since the projected image resolution is calculated projected image resolution = viewport * specified render resolution / camera viewport. The viewport (not camera viewport) is calculated from the modelView matrix.

Currently investigating a fix.

Viewport calc: https://github.com/BrownBiomechanics/Autoscoper/blob/794539cf53e79114091cc7cf3d7b08e1a78bd291/libautoscoper/src/Tracker.cpp#L776-L820

[NicerNewerCar]

The issue appears to have been an incorrect conversion from the viewport to pixels.

The terminology is a little confusing, but this is what they used in the backend.

Previously, the size of the projected image was calculated as follows: projected image resolution = viewport * specified render resolution / camera viewport where viewport is the 2D projection of the volume onto the film plane.

Initially, I had assumed that the film plane (viewport) and the image plane (camera viewport) were the same (since from what I can tell these terms can be used interchangeably in computer graphics) however this is not the case in Autoscoper.

In Autoscoper, the film plane is placed 2mm directly behind the camera and the image plane is the location of the radiograph in 3D space (placed in front of the camera at a distance determined by the focal length). So calculating the projected image resolution is a function of film plane space and image plane space. However, to the best of my knowledge, the registration and optimization takes place on the film plane and not the image plane.

Here is a rough figure to help explain the geometry:

To convert from canvas coordinates to viewport coordinates we can use the following formula:

Assuming we are giving a point (x,y) on a canvas of size WxH.

(x,y) \rightarrow ( \frac{2x}{W} - 1, \frac{2y}{H} - 1)

Then we can flip this around to go from viewport coordinates back to canvas coordinates:

(\frac{W(x+1)}{2}, \frac{H(y+1)}{2}) \leftarrow (x,y)

I will be following up with a PR shortly.

[amymmorton]

What does canvas coord mean above? canvas == image plane == the radiograph in 3D space (in mm? pixels?)

Good stuff Anthony, nice catch Cesar and Dajung

[NicerNewerCar]

What does canvas coord mean above? canvas == image plane == the radiograph in 3D space (in mm? pixels?)

Good stuff Anthony, nice catch Cesar and Dajung

Canvas coordinates are in pixels, what I am calling the canvas is the rendered composite image (radiograph and drrs) used during the registration (what is labeled as the film plane in the figure above). Its size is specified by the render resolution keyword in the config file.

[amymmorton]

awesome- thank-you

On Wed, Sep 6, 2023 at 4:33 PM Anthony Lombardi @.***> wrote:

What does canvas coord mean above? canvas == image plane == the radiograph in 3D space (in mm? pixels?)

Good stuff Anthony, nice catch Cesar and Dajung

Canvas coordinates are in pixels, what I am calling the canvas is the rendered composite image (radiograph and drrs) used during the registration (what is labeled as the film plane in the figure above). Its size is specified by the render resolution keyword in the config file.

— Reply to this email directly, view it on GitHub https://github.com/BrownBiomechanics/Autoscoper/issues/203#issuecomment-1709065340, or unsubscribe https://github.com/notifications/unsubscribe-auth/APUUPFVXG2SOS4WYREZZG53XZDM2PANCNFSM6AAAAAA4GMZZWI . You are receiving this because you commented.Message ID: @.***>

--

Amy Morton, MSc

Sr. Research Engineer, Bioengineering Lab, Department of Orthopaedic Research The Warren Alpert Medical School of Brown University and Rhode Island Hospital 1 Hoppin Street, CORO West, Ste. 404, Providence, RI 02903

[amymmorton]

These are images saved during debug mod of a wrist trial.

The size of render_width and render_height impact the pixel dimensions and resolution of the drr and rad to go into ncc calculation. Rad dimensions 1760 x 1760 Input render resolution variable in cfg: 880 880

Original clipped rad dimensions: 155 x 207

post commit SAM rad dims: 483 x 498

The drr aspect ratio has changed as well.. though I'm not certain the post-commit aspect ratio is correct

[NicerNewerCar]

One issue I've found is that the changes I made in #204, are correct for a conversion from a viewport coordinate to a pixel coordinate. However, I am only doing this conversion for the max XY of the viewport and not taking into account the min XY (ie, the difference between these two coordinates). If we take the difference between the min and max of the viewport, we get better results but the aspect ratio still looks off to me. I've included below a take with a brief comparison performed on the radius of dataset WN00113.

	DRR	Rad	Mask	Image Size
PR #204				479x508
Proposed Changed (minMaxDiff)				45x160
Old method				127x240

[jcfr]

Let's introduce a setting for explicitly using the updated approach.

By default, the logic before the integration of https://github.com/BrownBiomechanics/Autoscoper/pull/204 should be used.

[amymmorton]

I remain skeptical about the proposed change to render_width and render_height. I'm not convinced that was the initial cause of the buffer error... it is- but it isn't

When debugging - I had disabled the initial call to optimize frame from AutoscoperMainWindow.. I brought up in a zoom call that this call is an issue. Trying to track/optimize without loaded filters or an initial guess is the cause of these errors.

With the reverted rend-Width and height I was able to load and track the BVR shoulder data without encountering the buffer issues

When debugging - I had disabled the initial call to optimize frame from AutWindowMain.. which means I'm not immediately trying to track a rigid body without settings (filters/positioned in a initial guess). If the optical center of the image/camera are not 'centered' in the image plane, the initial load location of the DRR will be "off the edge" outside the bounds of the radiograph.

If- once the radiograph and drr are loaded- BUT THE DRR IS NOT PLACED IN AN INITIAL GUESS IN THE RADIOGRAPH BOUNDS... the buffer error occurs. Additionally- if the DRR exceeds the bounds of the radiograph and the frame is optimized this issue occurs.

However- this is a necessary state we need to account for- but not by the proposed calculation.

Proposed corrections

• [ ] disable the optimized frame call when setting up the trial in AutMainWindow
• [ ] mutually dependent DRR<->rad size calculations.: Here's what I mean - The comparison(what's passed the NCC) buffer width and height need to have the following characteristics:
  • [ ] aspect ratio true to anatomy (proportional to the original DRR/rad (both should have same mm/pix resolution)
  • [ ] rad search space should be inherited from the viewport of the projected DRR bounding box (already done in calculate viewport) -
  • [ ] the DRR should not exceed the bounds of the radiograph (that is what is truly causing this error) - however- the real position of the DRR will need to be 'outside' the radiograph in some frames- so we need to trim it (mask) by the bounds of the rad too

[amymmorton]

@NicerNewerCar Is this applicable to our viewport calc ? :

https://www.geeksforgeeks.org/window-to-viewport-transformation-in-computer-graphics-with-implementation/

It appears to be, the calculated viewport in Tracker returns x, y width and height in normalized ccords,

values such as -0.39 -0.10 0.42 0.34 (rounded of course).

Which aligns with RadRenderer

So can qwe query the viewport min,min width hieght of the rad- compare to vp_drr and correct as needed?