Closed NicerNewerCar closed 11 months ago
After a quick initial investigation, it would appear that the data loads in fine. The error happens when the NCC is calculated for the first time.
Disabling the NCC calculation yields the following result:
While this loads in, I would not consider this "fine" as I mentioned above.
Inspecting this volume in Slicer, I cannot find anything out of the ordinary compared to other data (from the same dataset) that works fine.
My plan moving forward is to do a complete comparison of the rendering pipelines for the broken and working data.
Playing around with the X-ray intensity the volume actually looks fine. Maybe this has something to do with the error?
The error is caused when the projected image resolution is higher than the specified render resolution. For the above model that is 764x467
vs 512x512
. Simply increasing the specified render resolution won't suffice since the projected image resolution is calculated projected image resolution = viewport * specified render resolution / camera viewport
. The viewport
(not camera viewport
) is calculated from the modelView
matrix.
Currently investigating a fix.
Viewport calc: https://github.com/BrownBiomechanics/Autoscoper/blob/794539cf53e79114091cc7cf3d7b08e1a78bd291/libautoscoper/src/Tracker.cpp#L776-L820
The issue appears to have been an incorrect conversion from the viewport to pixels.
The terminology is a little confusing, but this is what they used in the backend.
Previously, the size of the projected image was calculated as follows: projected image resolution = viewport * specified render resolution / camera viewport
where viewport
is the 2D projection of the volume onto the film plane
.
Initially, I had assumed that the film plane
(viewport
) and the image plane
(camera viewport
) were the same (since from what I can tell these terms can be used interchangeably in computer graphics) however this is not the case in Autoscoper.
In Autoscoper, the film plane
is placed 2mm directly behind the camera and the image plane
is the location of the radiograph in 3D space (placed in front of the camera at a distance determined by the focal length). So calculating the projected image resolution
is a function of film plane space
and image plane space
. However, to the best of my knowledge, the registration and optimization takes place on the film plane
and not the image plane
.
Here is a rough figure to help explain the geometry:
To convert from canvas coordinates to viewport coordinates we can use the following formula:
Assuming we are giving a point (x,y) on a canvas of size WxH.
(x,y) \rightarrow ( \frac{2x}{W} - 1, \frac{2y}{H} - 1)
Then we can flip this around to go from viewport coordinates back to canvas coordinates:
(\frac{W(x+1)}{2}, \frac{H(y+1)}{2}) \leftarrow (x,y)
I will be following up with a PR shortly.
What does canvas coord mean above? canvas == image plane == the radiograph in 3D space (in mm? pixels?)
Good stuff Anthony, nice catch Cesar and Dajung
What does canvas coord mean above? canvas == image plane == the radiograph in 3D space (in mm? pixels?)
Good stuff Anthony, nice catch Cesar and Dajung
Canvas coordinates are in pixels, what I am calling the canvas is the rendered composite image (radiograph and drrs) used during the registration (what is labeled as the film plane
in the figure above). Its size is specified by the render resolution
keyword in the config file.
awesome- thank-you
On Wed, Sep 6, 2023 at 4:33 PM Anthony Lombardi @.***> wrote:
What does canvas coord mean above? canvas == image plane == the radiograph in 3D space (in mm? pixels?)
Good stuff Anthony, nice catch Cesar and Dajung
Canvas coordinates are in pixels, what I am calling the canvas is the rendered composite image (radiograph and drrs) used during the registration (what is labeled as the film plane in the figure above). Its size is specified by the render resolution keyword in the config file.
— Reply to this email directly, view it on GitHub https://github.com/BrownBiomechanics/Autoscoper/issues/203#issuecomment-1709065340, or unsubscribe https://github.com/notifications/unsubscribe-auth/APUUPFVXG2SOS4WYREZZG53XZDM2PANCNFSM6AAAAAA4GMZZWI . You are receiving this because you commented.Message ID: @.***>
Amy Morton, MSc
Sr. Research Engineer, Bioengineering Lab, Department of Orthopaedic Research The Warren Alpert Medical School of Brown University and Rhode Island Hospital 1 Hoppin Street, CORO West, Ste. 404, Providence, RI 02903
These are images saved during debug mod of a wrist trial.
The size of render_width and render_height impact the pixel dimensions and resolution of the drr and rad to go into ncc calculation. Rad dimensions 1760 x 1760 Input render resolution variable in cfg: 880 880
Original clipped rad dimensions: 155 x 207
post commit SAM rad dims: 483 x 498
The drr aspect ratio has changed as well.. though I'm not certain the post-commit aspect ratio is correct
One issue I've found is that the changes I made in #204, are correct for a conversion from a viewport coordinate to a pixel coordinate. However, I am only doing this conversion for the max XY of the viewport and not taking into account the min XY (ie, the difference between these two coordinates). If we take the difference between the min and max of the viewport, we get better results but the aspect ratio still looks off to me. I've included below a take with a brief comparison performed on the radius of dataset WN00113.
DRR | Rad | Mask | Image Size | |
---|---|---|---|---|
PR #204 | 479x508 | |||
Proposed Changed (minMaxDiff) | 45x160 | |||
Old method | 127x240 |
Let's introduce a setting for explicitly using the updated approach.
By default, the logic before the integration of https://github.com/BrownBiomechanics/Autoscoper/pull/204 should be used.
I remain skeptical about the proposed change to render_width
and render_height
. I'm not convinced that was the initial cause of the buffer error... it is- but it isn't
When debugging - I had disabled the initial call to optimize frame from AutoscoperMainWindow
.. I brought up in a zoom call that this call is an issue. Trying to track/optimize without loaded filters or an initial guess is the cause of these errors.
With the reverted rend-Width and height I was able to load and track the BVR shoulder data without encountering the buffer issues
When debugging - I had disabled the initial call to optimize frame from AutWindowMain.. which means I'm not immediately trying to track a rigid body without settings (filters/positioned in a initial guess). If the optical center of the image/camera are not 'centered' in the image plane, the initial load location of the DRR will be "off the edge" outside the bounds of the radiograph.
If- once the radiograph and drr are loaded- BUT THE DRR IS NOT PLACED IN AN INITIAL GUESS IN THE RADIOGRAPH BOUNDS... the buffer error occurs. Additionally- if the DRR exceeds the bounds of the radiograph and the frame is optimized this issue occurs.
However- this is a necessary state we need to account for- but not by the proposed calculation.
DRR<->rad
size calculations.:
Here's what I mean - The comparison(what's passed the NCC) buffer width and height need to have the following characteristics:
DRR/rad
(both should have same mm/pix
resolution)@NicerNewerCar Is this applicable to our viewport calc ? :
It appears to be, the calculated viewport in Tracker returns x, y width and height in normalized ccords,
values such as -0.39 -0.10 0.42 0.34 (rounded of course).
Which aligns with RadRenderer
So can qwe query the viewport min,min width hieght of the rad- compare to vp_drr and correct as needed?
This error has been reported by Cesar and Dajung
https://github.com/BrownBiomechanics/Autoscoper/blob/794539cf53e79114091cc7cf3d7b08e1a78bd291/libautoscoper/src/gpu/opencl/OpenCL.cpp#L992-L999