This PR lands significant improvements and expansions of the capabilities of menpo3d. Normally we would like to have smaller PR's introducing atomic units of work, but given the scale and interconnectedness of this work, and that we have basically been using this branch as defacto master for the last few months, I think we just need to bite the bullet and merge this onto master, then handle tidy up later on.
Summary of changes:
NICP
NICP is now a generator, so you can get back each iteration of the algorithm. Useful for visualization (not included in this PR)
NICP now let's you customise the data term and stiffness term per-vertex per-iteration.
NICP now performs AlignmentSimilarity between source and target when landmarks are provided
A warning is now printed when the non-suitesparse version of spspolve is used (as this is really slow!)
NICP allows you to provide a PCAModel for the source, which will then use for Active NICP (experiemental)
The verbose output from NICP is now cleaner.
This is all good work. The following problems exist though:
[ ] NICP documentation needs updating
[x] per-vertex data term is presently non-optional, and not validated, even though it should be
[x] ~Active NICP is presently not working as well as anticipated~ extracted active_non_rigid_icp - performance is still not as desired, but at least it's isolated now
[x] The code in general is in need of some cleanup
Software Rasterizer
A new CPU-based rasterizer is introduced. It has a more functional, simpler API when compared to the existing GLRasterizer (which continues to exist for now, unmodified).
For smaller images (640x480) and not-too-dense meshes (<100k verts) the CPU rasterizer is competitive in performance, as it does not have the issue of syncronising frame buffers off the GPU which is capped at the display refresh rate.
The CPU rasterizer is much less fiddly to get running - it doesn't require OpenGL/X
The majority of the CPU rasterizer is written in pure Python - only the depth buffering part of the algorithm was written in Cython for speed.
The focus of the software rasterizer is just calculating the barycentric coordinates and triangle indices for the visible pixel in an image for a mesh/camera
The camera is modelled as a menpo Transform - the rasterizer step expects as input the mesh in the pixel space.
There is no issue with the range of the depth buffer as it just uses float64 numbers in any range for the comparison
The following issues exist:
[x] barycentric coordinates and triangle indices are frequently passed around in pairs, and we are inconsistent in the present APIs about the ordering of the pairs. Need to fix this before we can merge. Note this will break internal people's private code who have been depending on this branch (from memory: @trigeorgis, @nontas, @patricksnape, @yuxiang-zhou)
3DMM Improvements
The 3DMM code has been updated to use the software rasterizer
This PR lands significant improvements and expansions of the capabilities of menpo3d. Normally we would like to have smaller PR's introducing atomic units of work, but given the scale and interconnectedness of this work, and that we have basically been using this branch as defacto master for the last few months, I think we just need to bite the bullet and merge this onto master, then handle tidy up later on.
Summary of changes:
NICP
AlignmentSimilarity
between source and target when landmarks are providedspspolve
is used (as this is really slow!)PCAModel
for the source, which will then use for Active NICP (experiemental)This is all good work. The following problems exist though:
active_non_rigid_icp
- performance is still not as desired, but at least it's isolated nowSoftware Rasterizer
GLRasterizer
(which continues to exist for now, unmodified).Transform
- the rasterizer step expects as input the mesh in the pixel space.float64
numbers in any range for the comparisonThe following issues exist:
3DMM Improvements
Camera
Other
.as_trimesh()
as_coloured_trimesh()
. We can remove these for now.I think all checkboxes need to be fixed before this can be merged. Further work we can do in seperate PRs.
Note that this PR is needed for the upcoming release of the LSFM 3DMM construction pipeline.