Priorities

[manzt]

Ok, I've been trying to go through the examples listed in niivue/ipyniivue#50 to understand the usage of APIs exposed by Niivue.

For the time being, our priorities are motivated by basic multi-planar, envisioning a Pythonic API for this:

volumes = [
  { "path": "../images/mini152.nii.gz", "colormap": "gray", "visible": True, "opacity": 1.0 },
  { "path": "../images/hippo.nii.gz", "colormap": "red", "visible": True, "opacity": 1 },
]
nv = Niivue(opts={ "sliceType": SLICE_TYPE.MULTIPLANAR })
nv.load_volumes(volumes)
nv # displays widget

Additionally, updating properties on the images should happen through getters/setters for the properties on the created volumes (rather than inconsistent nv.setColormap(volumeId, ...) or nv.setOpacity(index, ...) . For the viewer rendered above should update with:

nv.volumes[0].colormap = "blue"
nv.volumes[0].opacity = 0.2
nv.volumes[0].visible = False

This primarily focuses on Volume for now, but we will do a similar audit of features necessary for Meshes. The following are the TODOs:

Volumes

• [x] Add opts traitlet for all options (can later add methods to update opts specifically in Python) niivue/ipyniivue-experimental#4
• [x] Implement custom file serializer for binary data niivue/ipyniivue-experimental#4
• [x] Implement load_volumes with _volumes traitlet niivue/ipyniivue-experimental#4
• [x] Make nv.volumes[idx] reactive for opacity, colormap, ~visible~ niivue/ipyniivue-experimental#4

More reactive APIs like add_volume can be added on top of this infrastructure, but complicate implementing the essential features.

Meshes

Find a similar example to basic multi-planar and audit the necessary APIs.

Observations

I took a look in the demos/ directory and a very high-level summary of the APIs in use from nv instances:

rg -o 'nv\d?\.(?:opts\.)?\w+' | sed 's/.*:nv[0-9]\.//' | sort | uniq -c | sort -r

 229 meshes
 107 setSliceType
  96 setMeshLayerProperty
  88 volumes
  87 attachTo
  73 loadVolumes
  65 drawScene
  61 updateGLVolume
  57 opts.dragMode
  56 dragModes
  55 setMeshProperty
  50 sliceTypeMultiplanar
  48 setClipPlane
  46 opts.multiplanarForceRender
  44 setMeshShader
  44 opts.isColorbar
  42 moveCrosshairInVox
  41 setPenValue
  38 loadMeshes
  32 sliceTypeRender
  30 graph
  29 setRadiologicalConvention
  24 setSliceMM
  21 meshShaderNames
  20 opts.backColor
  19 setOpacity
  18 setInterpolation
  17 setHighResolutionCapable
  16 drawOpacity
  12 setFrame4D
  12 setDrawingEnabled
  12 scene
  12 opts.meshXRay
  11 setRenderAzimuthElevation
  11 drawFillOverwrites
  10 opts.show3Dcrosshair
  10 opts.isRadiologicalConvention
   8 getFrame4D
   7 sliceTypeAxial
   7 setDrawColormap
   7 saveScene
   7 drawUndo
   7 drawGrowCut
   6 setModulationImage
   6 overlayOutlineWidth
   6 loadMatCapTexture
   6 loadDrawingFromUrl
   6 addLabel
   5 sliceTypeSagittal
   5 sliceTypeCoronal
   5 setSelectionBoxColor
   5 opts.yoke3Dto2DZoom
   5 opts.isHighResolutionCapable
   4 setSliceMosaicString
   4 setMultiplanarPadPixels
   4 setMultiplanarLayout
   4 setDrawOpacity
   4 setColormapNegative
   4 saveImage
   4 saveHTML
   4 saveDocument
   4 resizeListener
   4 removeHaze
   4 opts.penValue
   4 opts.isSliceMM
   4 opts.isOrientCube
   4 opts.isNearestInterpolation
   4 loadFont
   4 loadConnectome
   4 isAlphaClipDark
   4 gl
   4 drawOtsu
   4 colormaps
   4 attachToCanvas
   3 setVolumeRenderIllumination
   3 setMeshThicknessOn2D
   3 setCrosshairWidth
   3 setClipPlaneColor
   3 removeVolumeByUrl
   3 opts.clipPlaneColor
   3 onDragRelease
   3 getDescriptives
   3 broadcastTo
   3 addVolumeFromUrl
   2 thumbnailVisible
   2 setGamma
   2 setCustomMeshShader
   2 setCrosshairColor
   2 setColormap
   2 setAtlasOutline
   2 setAdditiveBlend
   2 removeMesh
   2 overlayAlphaShader
   2 opts.sagittalNoseLeft
   2 frac2mm
   1 vox2frac
   1 syncWith
   1 setScale
   1 setRenderDrawAmbientOcclusion
   1 setDefaults
   1 setCornerOrientationText
   1 reverseFaces
   1 refreshDrawing
   1 processImage
   1 overlayOutlineAlpha
   1 opts.multiplanarPadPixels
   1 opts.isV1SliceShader
   1 opts.isRuler
   1 opts.isMeshXRay
   1 opts.isForceMouseClickToVoxelCenters
   1 onMouseUp
   1 mm2frac
   1 loadFreeSurferConnectomeFromUrl
   1 loadDocument
   1 getMediaByUrl
   1 generateLoadDocumentJavaScript
   1 drawPt
   1 drawFloodFill
   1 drawBitmap
   1 createOnLocationChange
   1 createEmptyDrawing
   1 closeDrawing
   1 backgroundMasksOverlays
   1 addMeshFromUrl
   1 addMesh
   1 addColormap

I would recommend making decisions about next steps based on the prevalence and usage of each of these APIs. In the context of Python, I don't know if many of them make sense and many are essentially aliases for nv.opts.<property> = x and re-rendering, which should be covered by any opts traitlet we have.

[manzt]

It is not clear how to implement lazy loading for local 4D data since I believe this relies on HTTP range requests.

[manzt]

Visualization

[manzt]

@kolibril13 please have a look closely at this. Note how infrequently addVolume/addMesh are called in real world examples compared to loadVolumes/loadMeshes.

I think adding all the volumes/meshes should be the priority, the other "add" APIs can be added later (as an optimization, as i mentioned above).

[kolibril13]

Great progress! I like your systematic approach, these charts give a good overview of the project! I was thinking that the next todos could be to investigate this example and adding

• [ ] al_min, cal_max,visibility

to class Volume(ipywidgets.Widget)

https://niivue.github.io/niivue/features/additive.voxels.html

var volumeList1 = [
        { url: "../images/mni152.nii.gz" },
        {
          url: "../images/narps-4965_9U7M-hypo1_unthresh.nii.gz",
          colormap: "red",
          cal_min: 2,
          cal_max: 4,
        },
        {
          url: "../images/narps-4735_50GV-hypo1_unthresh.nii.gz",
          colormap: "green",
          cal_min: 2,
          cal_max: 4,
        },
      ];

[hanayik]

It is not clear how to implement lazy loading for local 4D data since I believe this relies on HTTP range requests.

@manzt , niivue does support loading specific volumes from 4D data using loadFromFile. It's just controlled by the limitFrames4D property.

However, this only works for NIFTI files (when the code detects). If not NIFTI, then the entire data range is loaded

However, i'm not sure if loadFromFile is used in the jupyter notebook implementation. Are files served via a local python server, or is data communicated by other means?

[manzt]

Volumes are serialized to binary data and send in one shot to the front end (over a websocket). We are using the new NVImage(arrayBuffer, ...) API with the array buffer from the iamge.