Note: This issue is a placeholder for notes about the transparency support within the 3D Viewer. It was started after a discussion between @aherbert and @ctrueden
When rendering a customnode.CustomMesh the default appearance set in org.scijava.java3d.PolygonAttributes are setBackFaceNormalFlip(true) and setCullFace(PolygonAttributes.CULL_NONE). This is a change from the java3d default which will disable backface normal flip and use PolygonAttributes.CULL_BACK. This means that transparent mesh polygons appear wrong.
In addition transparent objects only appear transparent to objects that have already been rendered. This is a java3d default for speed optimisation. It can be corrected in two ways which will be described below.
This java plugin demo demonstrates this:
import java.util.List;
import org.scijava.java3d.PolygonAttributes;
import org.scijava.vecmath.Color3f;
import org.scijava.vecmath.Point3f;
import customnode.CustomMesh;
import customnode.CustomTriangleMesh;
import customnode.MeshMaker;
import ij.gui.GUI;
import ij.plugin.PlugIn;
import ij3d.DefaultUniverse;
import ij3d.Image3DUniverse;
import ij3d.ImageWindow3D;
public class Transparency_Demo implements PlugIn
{
@Override
public void run(String arg)
{
createUniverse("Transparency_Demo Backface Normal Flip (Default)", false, false);
createUniverse("Transparency_Demo No Backface Normal Flip", true, false);
createUniverse("Transparency_Demo No Backface Normal Flip + Backface Cull", true, true);
}
private void createUniverse(String title, boolean disableBackfaceNormalFlip, boolean backfaceCull)
{
Image3DUniverse univ = new Image3DUniverse();
univ.showAttribute(DefaultUniverse.ATTRIBUTE_SCALEBAR, false);
univ.show();
ImageWindow3D w = univ.getWindow();
GUI.center(w);
w.setTitle(title);
addPoint(univ, disableBackfaceNormalFlip, backfaceCull, -1.5f, 0, 0, new Color3f(1, 0, 0));
addPoint(univ, disableBackfaceNormalFlip, backfaceCull, 1.5f, 0, 0, new Color3f(0, 1, 0));
addPoint(univ, disableBackfaceNormalFlip, backfaceCull, -1.8f, -0.3f, -1.5f, new Color3f(0, 0, 1));
univ.sync(true);
}
private void addPoint(Image3DUniverse univ, boolean disableBackfaceNormalFlip, boolean backfaceCull, float x,
float y, float z, Color3f c)
{
List<Point3f> points = MeshMaker.createIcosahedron(0, 1f);
for (Point3f p : points)
{
p.x += x;
p.y += y;
p.z += z;
}
CustomMesh mesh = new CustomTriangleMesh(points, c, 0.5f);
mesh.getAppearance().getPolygonAttributes().setBackFaceNormalFlip(!disableBackfaceNormalFlip);
if (backfaceCull)
mesh.getAppearance().getPolygonAttributes().setCullFace(PolygonAttributes.CULL_BACK);
univ.addCustomMesh(mesh, "Point " + x);
}
}
The default mode is shown below:
Backface normal flip is disabled:
Here the rendering improves as the icosahedron has been constructed with triangle facets that have normals that all face outwards from the center. Note that due to the lighting model under some orientations it is still possible to see through a polygon to the backface. This is best viewed by running the demo.
Backface normal flip is disabled and the culling is set to back face then the rendering is correct:
Order Dependent Transparency
However note that objects have been added to the scene as Red, Green then Blue. It is possible to view through the Blue object to both Red and Green; through the Green object to Red but not Blue; and impossible to view through the Red object.
There are two solutions to this transparency issue:
Sort all the objects before placing them on the scene from the viewpoint position. If the viewpoint changes then the sort must be redone and the objects reordered.
Use the java3d support for sorting transparent objects.
Option 1
Sorting the objects can be done using their depth to the current view point. This can be obtained as:
Transform3D ipToVWorld = new Transform3D();
univ.getCanvas().getImagePlateToVworld(ipToVWorld);
Point3d eyePtInVWorld = new Point3d();
univ.getCanvas().getCenterEyeInImagePlate(eyePtInVWorld);
ipToVWorld.transform(eyePtInVWorld);
All objects (e.g. using the object centre) can compute their distance to the eye point. Note: This can only be done if each object has no additional transformations applied beyond the virtual world, for example in the ImageJ3DViewer it is possible to rotate a CustomContent relative to the universe if the content transformation is not locked. If this is the case then the local transformation of objects must also be considered.
Once the distance to the eye point is known the rendering order can be updated. For any CustomMesh constructed using multiple objects this would involve reordering the coordinates of the mesh. For several CustomMesh objects added to the same scene then there is no simple solution as each mesh is added as separate content and the rendering order cannot be changed.
Option 2
Enabling the java3d support requires the view is configured:
Java3d will then sort individual geometries of each Shape3D within the scene graph.
By default this sorts objects using the centre-of-mass of their vertices, which is computed and cached. Optionally the runtime parameter -Dj3d.sortShape3DBounds=true allows "the bounds of the Shape3D node [to] be used in place of the computed GeometryArray bounds for transparency sorting for those Shape3D nodes whose boundsAutoCompute attribute is set to false" (see org.scijava.java3d.MasterControl.sortShape3DBounds). This allows better performance when the bounds of a shape can be computed at construction.
The documentation states that "Note that this policy will not split geometry into smaller pieces, so intersecting or intertwined objects may not be sorted correctly." Thus this option is only useful for objects that are roughly spherical if overlapping.
Having tried to use this solution within the ImageJ3DViewer I found it did not work. This is because the sorting is only applied to shapes that are not part of an ordered group (which has a prescribed rendering order), or any part of the graph below the ordered group. The default ij3d.ContentInstant that is added to the scene for all content uses an org.scijava.java3d.OrderedGroup to list the content node; the bounding box of the content; the coordinate system; and point list. Thus the order is fixed for content below this in the scene graph preventing dynamic sorting.
This has been fixed by using a CustomContentInstant that uses a org.scijava.java3d.Group to list the contents. The content node can then be constructed as a collection of org.scijava.java3d.Shape3D objects added to a org.scijava.java3d.Group. This can be placed onto the scene graph and sorted dynamically by the java3d framework.
Application
This is used to display super-resolution microscopy images within the 3D viewer. The super-resolution image is a results list of XYZ points. Each XYZ point must be drawn using a shape to show an image. Additional information about the points may be known, for example the precision of the XYZ locations. This data can be used to change the properties of the shape (e.g. size, colour, transparency).
The example below shows a dynamically rendered image of microtubules:
The localisations have been rendered using per localisation size dependent on the XY precision and the transparency set at 30%. Additionally per-item transparency is set using the localisation precision, hence larger objects that are more transparent have poorer precision. Due to the use of transparency the regions that are dense with localisations appear more saturated. Thus the microtubules can be seen to run parallel to each other. Note: These microtubules are on top of each other in the Z-axis and the z-resolution is poor.
A close up with high resolution spheres demonstrates the value of transparency:
The same microtubules can be rendered as a 2D image using circles:
Again the localisations have been rendered using per localisation size dependent on the XY precision and the transparency set at 80%. Additionally per-item transparency is set using the localisation precision, hence larger objects that are more transparent have poorer precision. A random dither is applied to the z-component so that overlapping objects combine via transparency to be more opaque. In this case the two microtubules can be seen to be parallel and approximately on-top of each other. Low precision results (large circles) are far away from the microtubules.
The dynamic sort performed by java3d is computationally expensive. The framework maintains a list of all transparent content within the scene, and caches the object centre. If the eye point is moved then the centre of each object is transformed back from the relative transformation within the scene graph (virtual world coordinates) to the image plate world coordinates. These are then compared to the eye point to get a distance. This requires a transformation per object but allows support for objects that are positioned using transformations.
Note that this is different to the suggested implementation above for option 1 which is to transform the eye point to the virtual world, requiring only one transformation for all objects. This is only applicable if all the objects have the same relative transformation within the scene graph, i.e. the actual coordinates of each object are different (and not controlled by transformations).
The ordered list of transparent objects is then updated using the new distances before rendering. This happens for each render frame.
Performance
I have found that the impact on rendering performance is noticeable when approximately 20,000 transparent objects are within the scene. Below this is is possible to move the view without any appreciable effect, allowing full transparency support to be used to view small super-resolution datasets.
Note: This issue is a placeholder for notes about the transparency support within the 3D Viewer. It was started after a discussion between @aherbert and @ctrueden
When rendering a
customnode.CustomMeshthe default appearance set inorg.scijava.java3d.PolygonAttributesaresetBackFaceNormalFlip(true)andsetCullFace(PolygonAttributes.CULL_NONE). This is a change from the java3d default which will disable backface normal flip and usePolygonAttributes.CULL_BACK. This means that transparent mesh polygons appear wrong.In addition transparent objects only appear transparent to objects that have already been rendered. This is a java3d default for speed optimisation. It can be corrected in two ways which will be described below.
This java plugin demo demonstrates this:
The default mode is shown below:
Backface normal flip is disabled:
Here the rendering improves as the icosahedron has been constructed with triangle facets that have normals that all face outwards from the center. Note that due to the lighting model under some orientations it is still possible to see through a polygon to the backface. This is best viewed by running the demo.
Backface normal flip is disabled and the culling is set to back face then the rendering is correct:
Order Dependent Transparency
However note that objects have been added to the scene as Red, Green then Blue. It is possible to view through the Blue object to both Red and Green; through the Green object to Red but not Blue; and impossible to view through the Red object.
There are two solutions to this transparency issue:
Option 1
Sorting the objects can be done using their depth to the current view point. This can be obtained as:
All objects (e.g. using the object centre) can compute their distance to the eye point. Note: This can only be done if each object has no additional transformations applied beyond the virtual world, for example in the ImageJ3DViewer it is possible to rotate a CustomContent relative to the universe if the content transformation is not locked. If this is the case then the local transformation of objects must also be considered.
Once the distance to the eye point is known the rendering order can be updated. For any
CustomMeshconstructed using multiple objects this would involve reordering the coordinates of the mesh. For severalCustomMeshobjects added to the same scene then there is no simple solution as each mesh is added as separate content and the rendering order cannot be changed.Option 2
Enabling the java3d support requires the view is configured:
Java3d will then sort individual geometries of each Shape3D within the scene graph.
By default this sorts objects using the centre-of-mass of their vertices, which is computed and cached. Optionally the runtime parameter
-Dj3d.sortShape3DBounds=trueallows "the bounds of the Shape3D node [to] be used in place of the computed GeometryArray bounds for transparency sorting for those Shape3D nodes whose boundsAutoCompute attribute is set to false" (seeorg.scijava.java3d.MasterControl.sortShape3DBounds). This allows better performance when the bounds of a shape can be computed at construction.The documentation states that "Note that this policy will not split geometry into smaller pieces, so intersecting or intertwined objects may not be sorted correctly." Thus this option is only useful for objects that are roughly spherical if overlapping.
Having tried to use this solution within the ImageJ3DViewer I found it did not work. This is because the sorting is only applied to shapes that are not part of an ordered group (which has a prescribed rendering order), or any part of the graph below the ordered group. The default
ij3d.ContentInstantthat is added to the scene for all content uses anorg.scijava.java3d.OrderedGroupto list the content node; the bounding box of the content; the coordinate system; and point list. Thus the order is fixed for content below this in the scene graph preventing dynamic sorting.This has been fixed by using a CustomContentInstant that uses a
org.scijava.java3d.Groupto list the contents. The content node can then be constructed as a collection oforg.scijava.java3d.Shape3Dobjects added to aorg.scijava.java3d.Group. This can be placed onto the scene graph and sorted dynamically by the java3d framework.Application
This is used to display super-resolution microscopy images within the 3D viewer. The super-resolution image is a results list of XYZ points. Each XYZ point must be drawn using a shape to show an image. Additional information about the points may be known, for example the precision of the XYZ locations. This data can be used to change the properties of the shape (e.g. size, colour, transparency).
The example below shows a dynamically rendered image of microtubules:
The localisations have been rendered using per localisation size dependent on the XY precision and the transparency set at 30%. Additionally per-item transparency is set using the localisation precision, hence larger objects that are more transparent have poorer precision. Due to the use of transparency the regions that are dense with localisations appear more saturated. Thus the microtubules can be seen to run parallel to each other. Note: These microtubules are on top of each other in the Z-axis and the z-resolution is poor.
A close up with high resolution spheres demonstrates the value of transparency:
The same microtubules can be rendered as a 2D image using circles:
Again the localisations have been rendered using per localisation size dependent on the XY precision and the transparency set at 80%. Additionally per-item transparency is set using the localisation precision, hence larger objects that are more transparent have poorer precision. A random dither is applied to the z-component so that overlapping objects combine via transparency to be more opaque. In this case the two microtubules can be seen to be parallel and approximately on-top of each other. Low precision results (large circles) are far away from the microtubules.
The dynamic sort performed by java3d is computationally expensive. The framework maintains a list of all transparent content within the scene, and caches the object centre. If the eye point is moved then the centre of each object is transformed back from the relative transformation within the scene graph (virtual world coordinates) to the image plate world coordinates. These are then compared to the eye point to get a distance. This requires a transformation per object but allows support for objects that are positioned using transformations.
Note that this is different to the suggested implementation above for option 1 which is to transform the eye point to the virtual world, requiring only one transformation for all objects. This is only applicable if all the objects have the same relative transformation within the scene graph, i.e. the actual coordinates of each object are different (and not controlled by transformations).
The ordered list of transparent objects is then updated using the new distances before rendering. This happens for each render frame.
Performance
I have found that the impact on rendering performance is noticeable when approximately 20,000 transparent objects are within the scene. Below this is is possible to move the view without any appreciable effect, allowing full transparency support to be used to view small super-resolution datasets.