MRView: Lazy loading of 3d textures

[rtabbara]

Currently, loading of 3d textures onto the GPU for images with a relatively high resolution can be quite an expensive operation, which in some instances can cause MRView to hang for a bit.

I was wondering whether it was worth allowing these 3d textures to be loaded incrementally per draw call rather than all at once. To add support for the progress bar, Image::copy_texture_3D is already designed in a way that partitions the uploading of the texture data, so enabling lazy loading wouldn't be too drastic a change.

I also thought it might be cool in the Volume mode to see the image built up as you loaded the 3d texture for the first time.

[draffelt]

A whole brain 3D progress bar...I like it.

[jdtournier]

A whole brain 3D progress bar...I like it.

:neckbeard:

I've also thought about incremental updates (although in quite a different way from this) but never really convinced myself that it could be done seamlessly. The problem as I see it is that the overwhelming majority of the time, you'll be loading the 3D texture for a generic tilted slice view. Having an incremental update might feel really weird in this case - and might actually jar even more. For example, if you happen to be displaying an axial slice, and just tilt the viewing plane slightly, that will trigger the upload of the 3D texture. If you do it incrementally, there will be no screen update until the data for that slice starts to come in, which may be halfway through or worse - during that time, there will be no visual feedback about this at all.

Also, while it would be fun to watch the 3D volume fill up, remember that a 3D volume render will generally be an expensive operation, especially for less-than-great hardware. So you might find that the load time grows massively in that case due to large number of renders during the load.

This issue isn't actually limited to the volume render either. You might be displaying render-intensive stuff (e.g. OFD overlay, lots of streamlines - especially with geometry shader now merged :wink: ), so again, load times might grow massively if the time taken per render is larger than the time to load a slice of data...

What should be happening when load times exceed half a second (I think) is that a progressbar will pop up to provide visual feedback as to what is going on. If that's not working, I'd rather we fixed that...

[rtabbara]

For example, if you happen to be displaying an axial slice, and just tilt the viewing plane slightly, that will trigger the upload of the 3D texture.

Yep, right now the 3d texture is incrementally built up along the z-axis, so what we could do is build the texture around a neighbourhood of the current focus (i.e. 0, +1, -1, +2, -2, etc. increment of the current V[2] voxel value). That way in the case of an axial slice you would immediately get visual feedback when moving from a 2d to 3d texture.

For the cases of sagital/coronal slices, if we wanted to be even fancier we could adjust the axis used to load the texture accordingly, so that again the entire texture would immediately be viewable.

Another benefit that might sway you is that this would be a nice way to handle volume changes when using a 3d texture --- in particular I noticed that when using the screen capture tool to animate volume changes, each frame would have to reload the entire 3d texture making the previewing a pretty choppy experience. In the lazy approach, you would only load the slice frame you're interested in (probably also need (edit) +n,-n for interpolation).

Also, while it would be fun to watch the 3D volume fill up, remember that a 3D volume render will generally be an expensive operation, especially for less-than-great hardware. So you might find that the load time grows massively in that case due to large number of renders during the load.

Well, if we incorporate the above suggestion of loading from the current focus, then the issue of waiting for visual feedback when viewing slices goes away. In the case of a volume, you're seeing everything anyway so you're going to always see part of the texture. Also, I'm not too sure that uploading a single slice per draw call is going to be a huge added burden on the pipeline. In the case of a pure 2d texture, when traversing slices we're constantly recreating the texture and that doesn't seem to cause any noticeable slowdown even when overlaying our rendering with other tools.

If I haven't wooed you then that's fine. But when grant season comes around and you need a cool visualisation of a rotating brain that builds up before your eyes, think about what you missed out on.

[jdtournier]

Well, it's not that I don't think it would be fun, but I don't see how you could implement that in a way that doesn't slow things down dramatically, particularly for the worst-case scenarios (expensive frame updates). And I also don't see that it would be trivial to implement lazy loading in a way that preferentially loads the slice displayed without massive overhead to support the worst-case scenario there (strongly oblique slices). I also see this might cause issues for the screen capture unless handled properly and very differently for that case. To expand:

• the issue is not whether loading a slice slows down the render - that is indeed negligible in most cases. The issue is that if you have many slices to load and you do one slice per render call, the render itself has the potential to become the bottleneck. For example, if I'm using the volume render with lots of tracks already loaded, and I open another large image, it will immediately be loaded as a 3D texture. If each screen update takes 0.1s say (so I'd normally get 10 fps), and I have 100 slices to load, it will take 10s to load the entire volume... to save maybe 0.5s on the direct upload as is currently done. I don't think it's worth it personally.
• it's actually really difficult to figure out which bits of the dataset might need to be loaded for the general case of an oblique slice, and keep track of what you've loaded. In the worst case, the slice might span all 3 axes equally, so you'd have to load a selection of voxels within ~1 voxel of the slice plane, and on the next update, figure out which you've already loaded and load only those that need to be updated - and have some mechanism for determining when everything has actually been loaded so you don't spent an enormous amount of time each frame to figure which bits have and haven't been loaded... You can probably use the approach of loading centre-out, which would at least let you use a predictable algorithm for the load, but personally I think this would get in the way when cycling through the volumes of e.g. a DWI dataset - a very common operation, and one that currently isn't particularly slow for a typical size dataset. I'm not really in favour of an approach that doesn't seamlessly display the slice you want to see, it would mean you have to cycle through volumes one by one, and wait for each to upload fully to make sure you haven't missed anything before moving onto the next - a productivity killer, I just want to hold down the arrow key to quickly inspect the data...
• This would also need to be handled differently for the screen capture: you need to make sure each frame is complete before grabbing it. If you do a render per slice load, you might have to wait quite a while before each frame is ready to be grabbed if the volume changes, especially for an expensive render (as explained in the first bullet point). This means your little movie clip could take an order of magnitude longer to produce... I guess you could somehow bypass the gradual upload when using the screen capture tool, but that would mean allowing a (conceptually optional) tool to influence core functionality, which starts to introduce non-trivial dependencies, and is something I'd consider not great software design...

So all in all, I think this would actually create many more problems than it solves... It's really not trivial to get right. Maybe others have opinions on the matter...?

[rtabbara]

@jdtournier Fair enough. I agree that it could get messy pretty quickly and for all the work to get it right, it's probably not worth the reward. Might consider revisiting this, but for now I'll close this off.