SIFT TDI grid artefact

[Lestropie]

Listing this here so that people know I'm aware of the issue and don't waste their time trying to explain it.

If SIFT is used to filter a tractogram, and a high-resolution TDI is then generated based on the filtered tractogram, there's an unusual grid-like artefact where the TDI is brighter along the diffusion voxel edges than through the middle of the voxels. It's particularly prominent anywhere where the streamlines run parallel to the diffusion voxel grid.

Some observations I've made in trying to find the source of this problem:

• Initially thought it might be related to the streamline-to-voxel mapping code, specifically the quantification of length through the voxel. E.g. if this quantification is wrong, a streamline that cuts through half of two voxels may appear longer than a streamline that cuts through all of one voxel. Upgraded the mapping code to do an accurate chordal approximation of length, and have quadruple-checked the mapping code for bugs (in particular making sure that if a streamline leaves and re-enters a voxel, both segments contribute).
• If the SIFT model cost function is changed from quadratic from linear, i.e. PM.(u.TD - FOD)^2 to PM.|u.TD - FOD|, the problem essentially disappears; this is however detrimental to the filtering process itself. Initially I thought this was related to the fact that the cost function change was calculated using a finite difference approach rather than partial derivatives, but changing to the latter didn't fix the problem either.
• If the streamline lengths through the voxels is ignored entirely (i.e. every streamline contributes unary length to every voxel it traverses), the problem is vastly reduced; but again, this is detrimental to the filtering.
• It's not related to the word-sharing used to reduce RAM usage in storing the streamline visitations.
• Resorting to an old-school Hermite upsampling of the streamline and counting points within each voxel rather than the voxel-edge mapping and length quantification doesn't solve the problem.
• Removing the scaling of the cost function gradient per unit streamline length before sorting the gradient vector doesn't solve the problem.
• Adding or subtracting streamline length from every voxel traversed (trying to find a bias toward streamlines encountering many voxel edges and therefore traversing more voxels) doesn't influence the artefact.

[Lestropie]

So it turns out this has nothing to do with the streamline mapping, or the quadratic v.s. linear cost function (that was an indirect solution). It's an effect of the SIFT algorithm itself, and the approximation it makes in order to make the execution time feasible.

So to solve it is going to require a fairly decent redesign of the sorting / filtering algorithm design, and will result in some amount of slowdown (I'll do my best to keep that to a minimum). But the eventual fix may affect anyone trying to use SIFT currently.

[draffelt]

How does it affect SIFT2?

[Lestropie]

Haven't done extensive testing, but what running I have done on the phantom, there's no artefact with SIFT2.

[Lestropie]

Removed release milestone: I had code close to achieving this, but it got deleted when I moved my local repositories around following the updated_syntax merge, and it's very much a non-trivial fix.