Status of this crate?

[RReverser]

I'm usually not the one to raise issues like this, but this is the only implementation of PnP I found in Rust, and I noticed that 1) this repo has never been published as a crate and 2) it hasn't been updated in a while to latest cv-core.

I wonder why - was a better alternative published elsewhere?

[vadixidav]

Absolutely no worries, I am always available to give comment to Rust CV crates.

In the cv mono repo, the PnP algorithm is currently implemented directly here in cv-sfm: https://github.com/rust-cv/cv/blob/511024feaa077a9af377cca7b654ad3d57d3bd6a/cv-sfm/src/lib.rs#L1452. I have been fairly busy recently, and so I have not been actively working on this right now, but at some point this would be refactored out into a proper pnp crate. In my mind, this crate is not complete. Looking at what I had written, it doesn't seem that the Levenberg-Marquardt optimization step was actually implemented.

Ultimately, this repository should be archived, and the pnp crate would need to be added to the cv repository. The way that PnP is implemented in cv-sfm is that you run the 2d to 3d correspondences through sample consensus using a similar process to that used in this repository, and then you follow it up with an optimization process. Currently, cv-sfm's optimization process only works when the match is already quite close, but that usually turns out to be fine. It has other issues when solving 3 view constraints, which it does for graph factorization, which it also poorly implements. A lot of work needs to be done on the SfM process implemented by Rust CV.

If you would like to perform pnp properly, I would recommend combining the arrsac, sample-consensus, cv-core, pinhole, lambda-twist, and levenberg-marquardt crates. arrsac is the consensus algorithm (a better alternative to RANSAC). sample-consensus is an abstraction around random sample consensus algorithms (which include RANSAC). lambda-twist is an Estimator for sample-consensus, and it is a fantastic P3P algorithm that can be used to implement PnP. cv-core contains the primitive types used for the camera model and in lambda-twist, and is designed to be used with sample-consensus. The pinhole crate contains the model for pinhole cameras. levenberg-marquardt is a crate we maintain (which is up-to-date on crates.io) which can optimize the resulting model using your inliers. It will minimize the L2 distance of your error criteria, but it only works for linear models. To use it you will have to derive the Jacobian matrix yourself. I don't currently have the Jacobian matrix derivation on hand, but if you do make it, I would really appreciate it if you could share the code since I could make use of that in cv-sfm and hopefully add it to an eventual pnp crate.

If you have any more questions or need any help with this, I am always available on Discord. Just leave a message in the general channel on the Rust CV Discord. You can find the link to the Discord server in the README as a badge.

[RReverser]

If you would like to perform pnp properly, I would recommend combining the arrsac, sample-consensus, cv-core, pinhole, lambda-twist, and levenberg-marquardt crates.

I actually started looking in that direction (of manually combining crates) shortly after raising this issue, but instead of lambda-twist I was looking into the p3p crate, merely because I found it (from dependencies of this pnp crate) but haven't seen the lambda-twist one. Descriptions of both crates sound very similar - what are the differences?

More generally, even though I'm developing intuition as to how this all should work, I'm by no means experienced in CV and don't have the necessary math knowledge, so this might take a bit longer for me to implement 😅

The other problem I have is that the focal length is not known in my case, but I suppose I can just extract the scale from the resulting inliers after finding the camera pose via ARRSAC & co..

[RReverser]

Another thing I was looking into was either porting SQPNP to Rust or just using it via Bindgen for now. It seems promising, especially if combined with RANSAC/ARRSAC.

[mpizenberg]

p3p was started for my specific needs (https://matthieu.pizenberg.fr/projects/rust-p3p/) and was intended to be as lean as possible, so it never got integrated with the types of the rust-cv other crates. That's why lambda-twist was started, to fill that gap. p3p is rather feature-complete and doesn't really need maintenance so I'm not spending time maintaining it as I don't have much time. Improvement could still be done once simd is better integrated into rust, but you'd have better chance using lambda-twist for a maintained and integrated crate.

[RReverser]

Hmm, should I be worried that out of the crates above cv-pinhole and lambda-twist are also marked as archived? Nvm, I see it's because they now live in the monorepo. Perhaps this should be added to the README of those repos.

[RReverser]

The other problem I have is that the focal length is not known in my case, but I suppose I can just extract the scale from the resulting inliers after finding the camera pose via ARRSAC & co..

Hmm, upon reading further I guess that's not enough and I need a P4P or P3.5P solver?

[mpizenberg]

The lambda twist crate has been merged into the CV monorepo

[vadixidav]

Also, I probably should have clarified, but I recommend directly pulling the crates in from the mono repo where applicable. I haven't updated the ones on crates.io in some time. Please reach out on Discord if you want help faster, but feel free to keep commenting on here.

[pinkforest]

If they are not supposed ot be used then perhaps yanking them would be best option to ensure people don't use them and fetch from the monorepo if intended to be used from there ?Thanks

[mpizenberg]

Yanking is an extreme measure, to use only when there is an important reason like a security threat or similar. This is a problem to be solved via documentation, not yanking.

[pinkforest]

Unmaintained crate is a security threat for some people.

How can you update documentation in crates.io - certainly you could upload a new release there to do that,.

But then it becomes maintained there again.