Using opencv's solvePNP or calibrateCamera for camera matrix estimation

[sreejith-mohanan]

Hi, it is possible to do the camera estimation with opencv's calibrateCamera or solvePNP? I was testing the same but i get different rotation and translation when i use them.

[patrikhuber]

Hi, Sure, it doesn't matter how you get the pose estimate. Depending on your application, you might need to decompose the matrix obtained from solvePnP to get the OpenGL rendering parameters, but maybe you don't need that. The pose estimate you get with different algorithms would always differ by a few degrees / pixels, but of course it also depends on what convention you use to represent the angles (e.g. yaw-pitch-roll or other orders).

[sreejith-mohanan]

Thanks for that quick response. For some reason i can trying the camera matrix estimation in python using opencv. Unfortunately i am not able to get a 3x4 matrix with last row as (0,0,0,1). I get other values in this row. I don’t find and eigen equivalent in python too. Can you please show me how to do it with opencv with some of the existing functions??

[patrikhuber]

solvePnP solves a general PnP problem with perspective camera. You can't get an affine camera with last row (0, 0, 0, 1) from that.

Currently, the linear shape fitting needs an affine camera. I think I could make it work with orthographic too, but that wouldn't solve it for you yet. However, if you want to do shape fitting with a perspective camera, you can have a look at fit-model-ceres in the devel branch! There I actually implemented perspective camera and nonlinear shape fitting.

[Thanh-Binh]

@patrikhuber I can not compile this devel branch in Ubuntu 16.04. There is an error as follows. Do you have any idea? ################################## In file included from /home/binh/Faces/eos/include/eos/render/render.hpp:25:0, from /home/binh/Faces/eos/examples/generate-obj.cpp:21: /home/binh/Faces/eos/include/eos/render/detail/render_detail.hpp: In function ‘void eos::render::detail::raster_triangle(eos::render::detail::TriangleToRasterize, cv::Mat, cv::Mat, boost::optionaleos::render::Texture, bool)’: /home/binh/Faces/eos/include/eos/render/detail/render_detail.hpp:533:109: error: invalid initialization of reference of type ‘const Vec2f& {aka const cv::Vec<float, 2>&}’ from expression of type ‘glm::tvec2<float, (glm::precision)0u>’ detail::tex2d(texcoords_persp, texture.get(), dudx, dudy, dvdx, dvdy); // uses ^ In file included from /home/binh/Faces/eos/include/eos/render/render.hpp:25:0, from /home/binh/Faces/eos/examples/generate-obj.cpp:21: /home/binh/Faces/eos/include/eos/render/detail/render_detail.hpp:298:11: note: in passing argument 1 of ‘cv::Vec3f eos::render::detail::tex2d(const Vec2f&, const eos::render::Texture&, float, float, float, float)’ cv::Vec3f tex2d(const cv::Vec2f& texcoords, const Texture& texture, float dudx, ^ examples/CMakeFiles/generate-obj.dir/build.make:62: recipe for target 'examples/CMakeFiles/generate-obj.dir/generate-obj.cpp.o' failed make[2]: * [examples/CMakeFiles/generate-obj.dir/generate-obj.cpp.o] Error 1 CMakeFiles/Makefile2:121: recipe for target 'examples/CMakeFiles/generate-obj.dir/all' failed make[1]: * [examples/CMakeFiles/generate-obj.dir/all] Error 2 Makefile:149: recipe for target 'all' failed make: *\ [all] Error 2

[sreejith-mohanan]

I am getting the same error too..

[sreejith-mohanan]

So i hope there isn’t ant predefined way to get an affine camera projection matrix using opencv? I was referring the doc and found the following

"In our approach we assume an affine camera and compute the normalised projection matrix, C̃ ∈ R3×4, using the Gold Standard Algorithm [107]. Given N ≥ 4 model to image point corre- spondences Xi ↔ xi , we determine the maximum likelihood estimate of C̃ which minimises: P i kx̃i − C̃X̃ik2 , subject to the affine constraint C̃3 = [0 0 0 1]"

can you please elaborate on the last line saying that minimizing P subject to the affine constraint C̃3 = [0 0 0 1]

[patrikhuber]

I'll fix the errors, thanks. It compiles under MSVC ;-) The project really needs CI for Linux.

@sreesreenu: I don't think OpenCV has an algorithm that estimates an affine camera, at least last time I checked. In any case the affine algorithm from H&Z that we used turned out to be not too suitable, so what fit-model does now is actually estimate an orthographic camera, and then convert the 4x4 ortho matrix into a 3x4 affine matrix.

I'm not sure what exactly you'd like me to elaborate on - an affine camera matrix has, I believe per definition, the last row [0, 0, 0, 1]. There's some more details about it in H&Z. But as I pointed out the orthographic camera estimate we use now is much more suitable, since the algorithm from H&Z actually allows non-uniform scaling and skew of the face model, which is not what we want in general.

[patrikhuber]

@sreesreenu, @Thanh-Binh: All compile errors in the devel branch on Linux are now fixed. Sorry for the issues and thanks for reporting!

[Thanh-Binh]

@patrikhuber How can I get the head position by using a perspective camera? Thx

[patrikhuber]

@Thanh-Binh, all you need should be in the fit-model-ceres example. Fitting perspective camera is not really different from orthographic camera.

[Thanh-Binh]

@patrikhuber Yes I used this example. What I believe is that a perspective camera can provide translation data (i.e head position) + and head rotation whereas orthographic ones only rotation. But in your example, I do not find any function or utility for getting the head position. Do I miss something? or are they not implemented?

[patrikhuber]

@Thanh-Binh: Orthographic camera provides t_x, t_y, scale, and rotation, whereas perspective camera provides t_x, t_y, t_z, focal_length, and rotation.

Have a look here on lines 236, 237, and after the fitting you can get the parameters from camera_translation_and_intrinsics (see e.g. L415) and camera_rotation L433/434.

[Thanh-Binh]

@patrikhuber thanks. It helps well. How can I get 3D-position of each LMs at the camera coordinate system?

[patrikhuber]

@Thanh-Binh: That's pretty easy as well:

auto model_view = t_mtx * rot_mtx; // like L414/415

This is the model-view matrix that transforms vertices from model space to view space (camera space), so just multiply this matrix (4x4) with any Morphable Model vertex (as homogeneous coordinates, 4x1 vector).

[Thanh-Binh]

understand, best thanks for your smart support

[sreejith-mohanan]

Thanks, i could finally complete my python implementation. I have used python lmfit to estimate the camera parameters, however the fitting is really slow. Can you suggest something else for the same.

Now regarding the previous comments about non-linear camera fitting, why not use a opencv's solvePNP to get the camera??

[patrikhuber]

@sreesreenu Do you know which part is slow? Always run a profiler. I think python's lmfit is really good - I don't know about the speed though. Maybe you want to try eos's python bindings? It's a bit experimental but will land in the devel branch soon. Bindings for fitting::estimate_orthographic_camera are already there, others can be added quite easily. That should give you the speed of the C++ implementation.

As mentioned before you can use OpenCV's solvePnP, but I believe it estimates a perspective camera, whereas I wanted orthographic so we can use it for the linear shape fitting (perspective introduces nonlinearity). And with solvePnP you have to decompose the resulting matrix to be able to render with OpenGL (or software), why I just estimate the rendering parameters directly. And I think under the hood solvePnP must use a nonlinear optimiser too, doesn't it?

You could also use solvePnP if you wanted to and then use the nonlinear shape fitting from fit-model-ceres - that would all work too.

[patrikhuber]

I'll close this, feel free to reopen if there are still questions about solvePnP.

In the meantime I've implemented a linear scaled orthographic camera estimation, which is the new default in fit-model in the master branch. It's really fast (~<1ms), maybe that's interesting for you.

[Hawk-TAO]

I want to get camera model matrix which can transform model space to camera space and use opengl to render. Is this way correct? static glm::vec4 v0(1, 0, 0, 0); static glm::vec4 v1(0, -1, 0, 0); static glm::vec4 v2(0, 0, -1, 0); static glm::vec4 v3(0, 0, 0, 1);

eos::fitting::RenderingParameters rendering_params = eos::fitting::estimate_orthographic_camera(image_points, model_points, srcImageCV.cols, srcImageCV.rows);
model = glm::mat4(v0, v1, v2, v3) * rendering_params.get_modelview();

but I can not get the correct matrix.

[Hawk-TAO]

And when will you complete the Perspective way?

[patrikhuber]

I don't think you need to multiply with glm::mat4(v0, v1, v2, v3), rendering_params.get_modelview() should already return a OpenGL conformant matrix.

And when will you complete the Perspective way?

Perspective camera estimation cannot be solved linearly, and it's actually ambiguous when only having a single image (if we don't know the camera sensor / focal length already). You can have a look at fit-model-ceres for a prototype of how perspective camera estimation can look like.

[Hawk-TAO]

hello, thank you in advance, I am a fresh in OpenCV and OpenGL, can you tell me why the Perspective camera estimation cannot be solved linearly? Is it because we don't know the camera sensor / focal length？ wait for your reply.Thank you.

At 2017-02-23 22:50:16, "Patrik Huber" notifications@github.com wrote:

I don't think you need to multiply with glm::mat4(v0, v1, v2, v3), rendering_params.get_modelview() should already return a OpenGL conformant matrix.

And when will you complete the Perspective way?

Perspective camera estimation cannot be solved linearly, and it's actually ambiguous when only having a single image (if we don't know the camera sensor / focal length already). You can have a look at fit-model-ceres for a prototype of how perspective camera estimation can look like.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[patrikhuber]

Perspective camera contains a division by z, which introduces nonlinearity. For more details I suggest you read a book like Hartley&Zisserman MVG, Shirley Computer graphics, and/or 3DMM literature.