How to set mapping from data space to object space when volume rendering

[Cody-G]

I'm combining volume rendering with particle rendering, and I would like to use a common coordinate system for both the particles and the volumes. By default it seem that any volume will be displayed within an object-space bounding box with corners at the origin and (1,1,1). So far I haven't found any way to change this setting. I've tried setting the dimension, dimensions, hull, boundingbox arguments when calling visualize (and also tried afterward with set_arg!) but it seems to have no effect on the rendering. Am I missing something?

I suppose I could also apply a transformation by setting model, but I would prefer a more direct method. What's the recommended way to do this? Thanks!

[SimonDanisch]

I guess this is a bug then. Let me look into it!

[Cody-G]

I also wouldn't be surprised if it's user error. I'm assuming that eyeposition is in the same coordinate space as hull and friends? Still working on it now, I'll let you know if I figure anything out.

[Cody-G]

There is another issue that may be unrelated: When I update the model for a volume rendering context, the transformation is applied to the 2D volume rendered image, not to the 3D volume before rendering as I expected. In order to say, rotate, the 3D volume and then rerender it (without calling visualize again), what should I do?

[SimonDanisch]

So you can change the dimension, which results in something like that:

I'm guessing you also want to move the origin?

[Cody-G]

Yes one thing I tried to do was set hull to a bounding box centered on the origin (so one corner has negative components).

[SimonDanisch]

I think I fixed most of the issues... let me test some last things and push them ;)

[Cody-G]

Awesome, thanks! That will really enhance a presentation that I'm giving tomorrow :)

[SimonDanisch]

Oh sweet! Would be nice to see what you're presenting!

[SimonDanisch]

Okay done: https://github.com/JuliaGeometry/GeometryTypes.jl/pull/78 https://github.com/JuliaGL/GLVisualize.jl/pull/157

After checking those out, you might also need to checkout GLAbstraction master, to be on the save side!

now you can scale/translate/etc via the model matrix, but you can also set the hull correctly! E.g:

visualize(volume, hull = AABB(Vec3f0(-1, -1, -1), Vec3f0(2, 2, 2))

for a volume that has it's origin at (-1, -1, -1), and is (2, 2, 2) wide!

[SimonDanisch]

If you have a powerful gpu and want to enhance the visualization fidelity, you can also play around with these in the source: https://github.com/JuliaGL/GLVisualize.jl/blob/sd/volume/assets/shader/volume.frag#L29

Sadly, they need to be compile time constants... But I could actually splice them in from the parameters to visualize...

[Cody-G]

Thanks Simon for such a fast response! This improved things by a lot! The hull argument works like I expect now. When I render the transformed volume it looks better than before, but I'm still getting a volume that looks thin when it is rotated. I assume this is because the ray tracing is going along an axis different from my viewing axis, and I'm only seeing the part of the volume that's superficial relative to the ray tracing axis (I'm using the :absorption method). I expected that the ray tracing angle would be determined by the light_position argument, but when I change the light position via set_arg! and re-render the volume looks the same. Is there another way to set the ray tracing axis, or to just keep it aligned with eyeposition?

Also, what happens when rendering multiple items that have different light_position or eyeposition settings? I am rendering both a volume and a set of particles. I suppose those and other arguments should apply only to the objects that they belong with?

[Cody-G]

And I'm happy to show you what I'm rendering! Really enjoying this package. I'll also play around with the shader settings for higher quality as you suggest :)

[SimonDanisch]

Ah yeah damn, rotation doesn't work -.- Too late yesterday to test. Let me look into it today!

[Cody-G]

Great, thank you! Also I have one more question. Is it possible to set the color for volume rendering? I have tried two methods that don't work:

1. Setting the color argument when rendering a Float32 array (my setting is ignored)
2. Rendering an RGBA or RGB data array (I see nothing rendered)

I've been rendering on a black background in case that makes a difference.

[SimonDanisch]

Color arrays are not implemented currently. You need to set:

color_map = RGBA{Float32}[....] # some color array
color_norm = Vec2f0(lower_limit, upper_limit) # defaults to extrema(volume)

[SimonDanisch]

Okay, rotation is hopefully fixed... I need better tests to verify if everything works nicely in all circumstances, though ;)

0fcb5b7aa78a3b7e04326c0c3365a80f685d4d37

[Cody-G]

Hey Simon, my video was really well-received at my seminar! Thanks for your great work. In the end I went with a particle visualization only. I'm still having trouble with rotations when volume rendering, and that commit doesn't seem to have changed things. Here is a stripped-down script demonstrating that the random volume appears hollow when you rotate it. I presume that's because the volume rendering isn't updating, or that it is updating with a ray tracing axis that doesn't match the eye position.

using GLVisualize, GeometryTypes, Reactive, GLAbstraction, Colors
using Quaternions
using Images

fish = rand(Float32, (221,241,139))
nsamps = 200
angles = linspace(0.0f0, 2*pi, nsamps)
res = (800,600)
window = glscreen(resolution=res, color=RGBA(0,0,0,0))
path = mktempdir()
name = "demo_vol_rot.mkv"
eyepos = Vec3f0(200)

datactr = Vec3f0(size(fish)...)./2f0
fishvol = visualize(fish, :absorption, absorption=0.1f0, hull=AABB{Float32}(-datactr, Vec3f0(size(fish))), light_position=Vec3f0(eyepos), light_intensity=Vec3f0(100f0))
_view(fishvol, window)
GLAbstraction.set_arg!(fishvol, :eyeposition, eyepos) 

#io, buffer = GLVisualize.create_video_stream(name, window)

for t = 1:nsamps
    GLAbstraction.set_arg!(fishvol, :model, transformationmatrix(Vec3f0(0), Vec3f0(1), qrotation([0.0f0;0.0f0;1.0f0], angles[t])))

    GLWindow.poll_reactive()
    #render stuff
    GLWindow.render_frame(window)
    GLWindow.swapbuffers(window)

    # add the frame from the current window
#    GLVisualize.add_frame!(io, window, buffer)

end

#close(io)
renderloop(window)

[SimonDanisch]

Glad and sad to hear! ;) oh...

GLAbstraction.set_arg!(fishvol, :eyeposition, eyepos) 

You shouldn't do that... let me see if that fixes it ;) What was your intention behind that?

[SimonDanisch]

Okay I got it! 3 problems, sorry for that! -.-

1) you need to change the eyeposition of the camera and not directly the eyeposition parameter. That is more of a hidden parameter, which gets set by the camera automatically, just not if you actually overwrite it ;) 2) I need the inverse of the model matrix, and since set_arg! is not implemented correctly currently, it doesn't actually update the inverse of the matrix, even though it should! 3) At least on my lame laptop, the asynchronous nature of OpenGL makes the rendering really stuttery. The loop goes through very quickly, while OpenGL lacks behind.

Here is an updated version which should work much better:

using GLVisualize, GeometryTypes, Reactive, GLAbstraction, Colors
using Quaternions

fish = rand(Float32, (221, 241, 139))
nsamps = 200
angles = linspace(0.0f0, 2f0*pi, nsamps)
res = (800, 600)
window = glscreen(resolution = res, color = RGBA(0,0,0,0))
datactr = Vec3f0(size(fish)...)./2f0
eyepos, lookat = Vec3f0(200), Vec3f0(0)
cam = PerspectiveCamera(window.inputs, eyepos, lookat)
model = Signal(eye(Mat4f0))
fishvol = visualize(
    fish, :absorption, absorption = 0.1f0,
    hull = AABB{Float32}(-datactr, Vec3f0(size(fish))),
    light_position = Vec3f0(eyepos),
    light_intensity = Vec3f0(100f0),
    model = model
)
_view(fishvol, window, camera = cam)
@async renderloop(window)
t = 1
m =
while isopen(window)
    push!(model, rotationmatrix_z(angles[mod1(t, nsamps)]))
    yield()
    ModernGL.glFinish() # not sure what to do about this. 
    # the async nature of OpenGL is usually desirable!
    t += 1
end
GLWindow.destroy!(window)

[SimonDanisch]

Btw, these kind of issues should become much better in future versions of GLVisualize. I have a much clearer picture of how the API should look like and because of that I hope to make the implementation much more coherent and understandable (e.g. the set_arg! stuff)!

[Cody-G]

Ah gotcha! Your script works well for me, thanks! If I want to add each frame to a video stream, I suppose I can still do that within the while loop? Should I do it after calling glFinish? I'm not sure what glFinish is doing.

I just assumed that changing eyeposition was equivalent to using the camera, oops! That also answers my other question about what happens if objects have different eyeposition settings. I guess it should never happen if they share the same camera.

One last thing: In the current design is it okay to use a Signal for any argument that could be passed to visualize for any kind of rendering? (not just volume rendering). Or are there still some arguments that can't be a Signal, and how would I know which?

I'm really enjoying playing with this, planning to do more over the next few days so that it's ready for my next presentation :)

[SimonDanisch]

I suppose I can still do that within the while loop?

Yes! glFinish waits for OpenGL to finish rendering, so it's a good idea to leave it there when recording a video.

Or are there still some arguments that can't be a Signal, and how would I know which?

Everything should support signals, but there are a few left which don't. Just open an issue about them and I'll fix them :)

Feel free to ask me anything before wasting your time!

[Cody-G]

Got it, thanks! Well you have solved my volume rendering issues for now so feel free to close this. I have a question about particle rendering that I'll put in a different issue.