Other transfer functions

OliverEvans96 commented 7 years ago

Hi @maartenbreddels,

I remember you expressing interest in other transfer functions. Currently, it seems that the TF with RGB gaussians is the only one implemented. I see that there are a few TransferFunction objects defined at https://github.com/maartenbreddels/ipyvolume/blob/master/ipyvolume/transferfunction.py.

I'm wondering if there's a clear entrypoint for implementing other transfer functions. Is it via modifying these objects I pointed out, or would some other infrastructure need to be rearranged? Can it be done via Python alone, or will it require JS?

Thanks! Oliver

maartenbreddels commented 7 years ago

Hi Oliver,

The current implementation that is used is that of [TransferFunctionWidgetJs3][https://github.com/maartenbreddels/ipyvolume/blob/771b76987e03800ddafa558f0518546d85b8094b/ipyvolume/transferfunction.py#L40] where all the work is being done on the js/browser side. However, the only thing that needs to be done is set the rgba trait of it's parent class (TransferFunction). You should be able to read out the rgba value of the Python side for the JS implementation, and assign this to a newly created transfer function, and play around with that. To be honoust I haven't tested this for a while, feel free to ask more or raise issues on this, happy to give extra pointers or do small fixes.

cheers,

Maarten

maartenbreddels commented 7 years ago

Just telling you know i'm testing things out, and transferring the .rgba trait is broken, so don't spend too much time on this yet :)

maartenbreddels commented 7 years ago

Ok, some fixes in master now.

Starting with a 3d plot like this:

import ipyvolume as ipv
import ipywidgets
import numpy as np
ds = ipv.datasets.aquariusA2.fetch()
fig = ipv.figure()
ipv.volshow(ds.data)
ipv.show()

We can now visualize the transfer function as image:

import PIL.Image
# * 255 for [0, 1] to [0,256) range
rgba = fig.tf.rgba * 255  # optionally multiply by ~4 to see the colors better
rgba = (np.repeat([rgba], 256, axis=0)) # repeat so we get a height of 256, instead of 1 pixel
print(rgba.shape)
rgba = rgba.astype(np.uint8)
im = PIL.Image.frombuffer("RGBA", rgba.shape[:2], rgba, 'raw')
im

We can now change the transfer function, by setting a new transfer function, or modifying the data directly (.rgba):

tf = ipv.transferfunction.TransferFunction(rgba=np.random.random((256, 4)))
fig.tf = tf
# or simply
fig.tf.rgba = np.random.random((256, 4))

Specific transfer function can subclass TransferFunction

OliverEvans96 commented 7 years ago

Thanks for your work on this, how cool!

I've created a widget interface for interactively editing the transfer function with bqplot's HandDraw.

I've submitted PR #76 with the notebook if you're interested!

I'm running into an issue where the transfer function doesn't seem to be set until the figure is actually rendered to the screen. And it seems to do it some times but not others. Have you experienced this? I was trying to automatically generate a figure, but I had to generate it, render it, and then pass it to the widget object in order to get it to work.

Cheers! Oliver

OliverEvans96 commented 7 years ago

Also, you've probably seen this since it's the first result that came up for me for "transfer function volume rendering", but I'll post it since it's relevant and it seems to have some good info on transfer functions. https://www.ics.uci.edu/~gopi/CS211B/TransferFunctions.pdf

maartenbreddels commented 7 years ago

I haven't seen this, it's quite interesting material, are you planning on trying to implement some of these techniques?

OliverEvans96 commented 7 years ago

I only skimmed it, but I'd like to learn a bit more and try to implement something.

On Fri, Oct 13, 2017, 3:14 PM Maarten Breddels notifications@github.com wrote:

I haven't seen this, it's quite interesting material, are you planning on trying to implement some of these techniques?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maartenbreddels/ipyvolume/issues/74#issuecomment-336543193, or mute the thread https://github.com/notifications/unsubscribe-auth/APLYIz1zNE4xOKUdfNvx4uwaB7QaERwmks5sr7atgaJpZM4PxcrR .

OliverEvans96 commented 7 years ago

I'm open to any ideas! My primary use case is for my Master's thesis, in which I'm visualizing probability distributions of seaweed and the light field around the seaweed.

I'm interested to see if my data is better suited to different types of transfer functions than your astronomical data, or whether the same basic ideas pretty much work well across the board.

On Fri, Oct 13, 2017, 3:16 PM Oliver Evans oliverevans96@gmail.com wrote:

I only skimmed it, but I'd like to learn a bit more and try to implement something.

On Fri, Oct 13, 2017, 3:14 PM Maarten Breddels notifications@github.com wrote:

I haven't seen this, it's quite interesting material, are you planning on trying to implement some of these techniques?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maartenbreddels/ipyvolume/issues/74#issuecomment-336543193, or mute the thread https://github.com/notifications/unsubscribe-auth/APLYIz1zNE4xOKUdfNvx4uwaB7QaERwmks5sr7atgaJpZM4PxcrR .

maartenbreddels commented 7 years ago

Interesting material, it might be a good to explore a few ways of auto generating transfer functions, or have diagostic plot like f' vs f to help designing one. One thing I'd like to have, is a good volume rendering of a head/skull, since I've only worked with non-realistic rendering. But there is great progress to make here, what is now in ipyvolume is quite rudimentary.

OliverEvans96 commented 7 years ago

Since the tf.rgba matrix is shape (256,4), we're restricted to 1D transfer functions (TF: R ->R^4). Multidimensional transfer functions (TF:R^n->R^4) take inthere's account not just data value, but other quantities such as data position (x,y,z) or gradient magnitude/direction, hessian norm, etc. It seems that these are better suited to edge detection between distinct homogeneous materials, as is the case in the human skull or tooth, for example.

One way to implement this would be to turn tf.rgba into shape (256,256,4), and have the second column be gradient magnitude. A more general way would be to provide an interface for directly setting rgba values at each voxel. (I'm not sure if this is already possible - probably from JS, but not Python?) Then, we could create a function which takes data values as an input, calculates whichever derivatives are of interest, and then sets the voxels directly as a function of the data values and derivatives.

On Fri, Oct 13, 2017, 3:33 PM Maarten Breddels notifications@github.com wrote:

Interesting material, it might be a good to explore a few ways of auto generating transfer functions, or have diagostic plot like f' vs f to help designing one. One thing I'd like to have, is a good volume rendering of a head/skull, since I've only worked with non-realistic rendering. But there is great progress to make here, what is now in ipyvolume is quite rudimentary.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maartenbreddels/ipyvolume/issues/74#issuecomment-336547243, or mute the thread https://github.com/notifications/unsubscribe-auth/APLYI75nvnNqNj3sY9Ydyq6dR8TARdMVks5sr7sLgaJpZM4PxcrR .

OliverEvans96 commented 7 years ago

Also: here's a thorough treatment of automatic TF generation: http://www.cs.utah.edu/~gk/papers/vv98/index.html

On Sat, Oct 14, 2017, 10:16 AM Oliver Evans oliverevans96@gmail.com wrote:

Since the tf.rgba matrix is shape (256,4), we're restricted to 1D transfer functions (TF: R ->R^4). Multidimensional transfer functions (TF:R^n->R^4) take inthere's account not just data value, but other quantities such as data position (x,y,z) or gradient magnitude/direction, hessian norm, etc. It seems that these are better suited to edge detection between distinct homogeneous materials, as is the case in the human skull or tooth, for example.

One way to implement this would be to turn tf.rgba into shape (256,256,4), and have the second column be gradient magnitude. A more general way would be to provide an interface for directly setting rgba values at each voxel. (I'm not sure if this is already possible - probably from JS, but not Python?) Then, we could create a function which takes data values as an input, calculates whichever derivatives are of interest, and then sets the voxels directly as a function of the data values and derivatives.

On Fri, Oct 13, 2017, 3:33 PM Maarten Breddels notifications@github.com wrote:

Interesting material, it might be a good to explore a few ways of auto generating transfer functions, or have diagostic plot like f' vs f to help designing one. One thing I'd like to have, is a good volume rendering of a head/skull, since I've only worked with non-realistic rendering. But there is great progress to make here, what is now in ipyvolume is quite rudimentary.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maartenbreddels/ipyvolume/issues/74#issuecomment-336547243, or mute the thread https://github.com/notifications/unsubscribe-auth/APLYI75nvnNqNj3sY9Ydyq6dR8TARdMVks5sr7sLgaJpZM4PxcrR .

maartenbreddels commented 7 years ago

I think the idea of doing a 2d transfer function indeed is a good idea. We can give it a shape of (N,M,4), where N is the f direction, and M the f' direction, and M can be of length 1 (basically a 1d transfer function).

A practical problem is how to store it, f(\vec{x}) is stored in the alpha channel, and the derivative (normalized since it is used for lighting only now) in RGB. We could store it unnormalized, but since it is just 8 bits (-128,127) the resolution wil be quite poor. Another solution would be to store the spherical angles for the normal in say the R and G channel, and the amplitude in B.

I'm not sure if i'm a fan of setting directly the rgba values, since that would make it impossible to change the transfer function interactively sincevolumetric datasets are often quite big, and transferring it can take a lot of time. Just updating the transfer function is much cheaper, and can happen on the JS side, meaning it does not need a running kernel.

OliverEvans96 commented 7 years ago

So you're talking about how to store the (N,M,4) transfer function? I think we can throw out the direction, actually. From what I've been reading, it seems that the magnitude is really all that matters. So then we have:

Dim 1: f(\vec{x}) Dim 2: ||\grad f(\vec{x})|| Dim 3: RGBA

And we have N = M = 256 (or M=1 for 2D) and all the entries are between 0 and 1.

I think I might be missing what you're getting at, though. If so, could you elaborate on the problem?

On Sat, Oct 14, 2017, 12:37 PM Maarten Breddels notifications@github.com wrote:

I think the idea of doing a 2d transfer function indeed is a good idea. We can give it a shape of (N,M,4), where N is the f direction, and M the f' direction, and M can be of length 1 (basically a 1d transfer function).

A practical problem is how to store it, f(\vec{x}) is stored in the alpha channel, and the derivative (normalized since it is used for lighting only now) in RGB. We could store it unnormalized, but since it is just 8 bits (-128,127) the resolution wil be quite poor. Another solution would be to store the spherical angles for the normal in say the R and G channel, and the amplitude in B.

I'm not sure if i'm a fan of setting directly the rgba values, since that would make it impossible to change the transfer function interactively sincevolumetric datasets are often quite big, and transferring it can take a lot of time. Just updating the transfer function is much cheaper, and can happen on the JS side, meaning it does not need a running kernel.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maartenbreddels/ipyvolume/issues/74#issuecomment-336647095, or mute the thread https://github.com/notifications/unsubscribe-auth/APLYIwBggQhdikF8-qliAg6qNCeK8Cnsks5ssOM1gaJpZM4PxcrR .

maartenbreddels commented 7 years ago

The direction (without magnitude) is needed for lighting effect, so for each voxel we need

||\grad f(\vec{x})|| (for the TF)
\grad f(\vec{x})/||(\grad f(\vec{x})|| (for lighting)
f(\vec{x}) (the data) which should fit in 4 channels, rgba (not the transfer function, the volumetric data texture), but i'm not yet sure what is the best way.

OliverEvans96 commented 7 years ago

Ah, okay, I think I'm understanding now. So we have two arrays: The volumetric data texture (VDT) and the transfer function (TF).

The VDT:

has shape (N_x, N_y, Nz, 4), where N is the size of the data in the direction.
VDT[i,j,k,:] describes f(x_i, y_j, z_k) and the gradient at the same point.
presumably contains 8 bit signed integers.

The TF:

has shape (N,M,4) where N = M = 256 for 2D TF, or N = 256, M = 1 for 1D TF.
TF[i,j,:] describes the RGBA values which will be assigned to a voxel with function value f_i and gradient value gradf_j, where f0 = min(f), f(N-1) = max(f), and gradf0 = min(grad f), gradf(M-1) = max(grad f)
contains floating point values between 0 and 1.

Then, the RGBA values assigned to the voxel at point (x_i, y_j, z_k) are: TF[ DTV[x_i, y_j, z_k, 0], DTV[x_i, y_j, z_k, 1], :] (assuming that for a particular point, f is stored in column 0 and the magnitude of the gradient is stored in column 1, and the other two columns are the spherical angles)

Am I understanding correctly?

maartenbreddels commented 7 years ago

👍 indeed, N and M don't have to be 256 btw.

This automatic TF paper is really interesting btw, and well written! Food for thought this weekend :) I am wondering, since it's from 98, what progress has been made since then. But I like the simplicity, although it focusses mostly on edge detection, which it maybe not something you always want.

OliverEvans96 commented 7 years ago

Cool. So why does storing the full vector produce more roundoff error than the cartesian components of its unit vector? Do you think that storing the gradient in spherical coordinates reduces the round-off error compared to unnormalized cartesian?

maartenbreddels commented 7 years ago

If we store the gradient unnormalized, and rescale between [-128, 127], small values of the gradient (small as in the norm is small, say in the range [0, 3]), the directions will be heavily discretized, giving non smooth lighting effects. If we store instead the spherical angles for the direction, it wil always have the same resolution and requires just 2 components, and we can use the 3rd component for the gradient norm (which is what we need for the TF anyway). So I think going to spherical coordinates for the gradient may be better. Hope that explains it a bit more.

cancan101 commented 7 years ago

I found this c++ implementation of the 98 paper on auto TF.

Here is the reference (I think) implementation, and these are some docs on using the tool.

ericwhester commented 6 years ago

Hello Maarten, thanks for this fantastic visualisation package! And your widget looks really useful Oliver. Changing the transfer function is the one thing I think IPyvolume could most benefit from.

It seems like the pull request didn't work? Is this correct? Or is there some way we could use your widget (Oliver) to alter the transfer function?

OliverEvans96 commented 6 years ago

@ericwhester - I'm not sure what happened there. It looks like we had an idea for an improvement, didn't get around to merging, and then I must have messed up and modified the branch, because I don't see the notebook there. Version control gone wrong...

Luckily, I had a copy on my computer, which I've put here: https://github.com/OliverEvans96/ipyvolume/blob/tf_notebook/notebooks/IPyVolume%20Transfer%20Functions.ipynb

The widget is a bit finnicky - sometimes you have to rerun the cell a few times. I couldn't tell you exactly why, but hopefully you'll get it to work!

Hopefully that will be enough to play with for now :) Oliver

OliverEvans96 commented 6 years ago

Whoops - I had the wrong link - fixed now.

ericwhester commented 6 years ago

Thanks for the quick reply Oliver. This is fantastic!

Two more (related) questions:

Is there a way to do a linear interpolation of the transfer function between specified points? So, you could add and subtract points which the r,g,b, and alpha channels linearly interpolate between. I've used transfer function editing in other software (such as vapor) which did this, and I think it worked well. Of course, anything interactive is a bit of a hassle, so no worries if that's too much to ask - I'm certainly not going to complain about the widget!

Also, if we had access to a desired transfer function (like an actual python callable object), what would be the best way to get this working with ipyvolume?

Thanks again!

OliverEvans96 commented 6 years ago

Sure thing!

Both should be doable. bqplot scatters have an enable_move keyword that allows points to be dragged and update other things. You can also restrict to x or y only if desired. I'm not 100% sure whether you can add points. I don't think I've tried it, but I'm sure there's a way.

As for setting the TF from a callable, that should be straightforward. You can see in the widget that all I'm doing is updating the ipv.figure.tf.rgbaarray. It looks like it's a 256x4 array with RGBA as columns. Note that as with most traitful things, you have to actually assign the array (e.g. rgba = np.random.rand(256,4)) rather than modify it (e.g. rgba[:,0] -= 1)

Oliver

OliverEvans96 commented 6 years ago

Scatter docs are here if you're interested in attempting it! http://bqplot.readthedocs.io/en/stable/_generate/bqplot.marks.Scatter.html

And examples: https://github.com/bloomberg/bqplot/blob/master/examples/Marks/Scatter.ipynb

maartenbreddels commented 6 years ago

Hi Eric,

thanks for the positive response! The transfer function definitely could use improvement, we lost a bit of momentum in that... I think we could start by transforming what Oliver made into a helper function in transferfunction.py to get something interactive in.

ericwhester commented 6 years ago

Thanks for the help guys, this is great. I'll have a go at defining a custom transfer function.

OliverEvans96 commented 6 years ago

Awesome! Let us know how it goes!

On Mon, May 7, 2018, 11:02 PM ericwhester notifications@github.com wrote:

Thanks for the help guys, this is great. I'll have a go at defining a custom transfer function.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/maartenbreddels/ipyvolume/issues/74#issuecomment-387269858, or mute the thread https://github.com/notifications/unsubscribe-auth/APLYI2k5YpUPxs0Jq6Nfc-88C7qzPiv8ks5twQrcgaJpZM4PxcrR .

OliverEvans96 commented 6 years ago

@ericwhester any luck here?

maartenbreddels commented 6 years ago

I would be nice to coordinate transfer functions, I know @vidarf was thinking about it, and @thewtex has a really nice widget in itk-jupyter-widgets, although that currently doesn't seem to be exposed to the kernel.

maartenbreddels commented 6 years ago

@OlehKSS might also be interested in this topic

thewtex commented 6 years ago

Here are the relevant sources used in itk-jupyter-widgets.

https://github.com/Kitware/vtk-js/blob/master/Sources/Interaction/Widgets/PiecewiseGaussianWidget/index.js

Unfortunately, they are pretty tied in with the rest of the code.

vidartf commented 6 years ago

@maartenbreddels What I'm thinking of is a simple colormap editor, similar to e.g. the gradient editor in Photoshop (https://www.photoshopessentials.com/basics/how-to-use-the-gradient-editor-in-photoshop/).

thewtex commented 6 years ago

@vidartf @maartenbreddels HTML now provides a nice color selector widget:

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/color

widgetti / ipyvolume

Other transfer functions #74