Please help with hand detection

[AdrianPeniak]

Hi,

I would like to kindly ask you for help with hands detection using mediapipe and tensorflow. I used your awesome project (forked here https://github.com/AdrianPeniak/mediapipe_face_iris_cpp.git) and port it to linux env. However when I reuse your classes for hand detection using palm model I can't get good results (hands not detected, drawing random roi according code in repo). I understand that functions as getHighestScoreDetection() etc. must be rewritten, but I afraid that problem is somewhere before (e.g. image preprocessing).

Could you please take a look to my forked project?

Thanks for any advise.

Thanks

[pntt3011]

Thank you for the issue and I am so sorry for my VERY late response. As for the palm detection model, I think the problem is how you generate the anchors for the model's output. Maybe your anchor sequence does not match the model's output. I've tried a customized model from this repo: https://github.com/aashish2000/hand_tracking and it works like a charm. Here is my solution:

1. Use the _palm_detection_without_customop.tflife instead of _palmdetection.tflite.
2. Move anchors.csv (also in that repo) to models folder.
3. Change create_ssd_anchors_hand() and its related functions to the following functions:

   
   #include <fstream>
   #include <sstream>
   #include <string>

cv::Rect2f convertAnchorVectorToRect(const std::vector &v) { float cx = v[0]; float cy = v[1]; float w = v[2]; float h = v[3]; return cv::Rect2f(cx - w / 2, cy - h / 2, w, h); }

std::vector generateAnchors() { std::vector anchors; std::ifstream file("./models/anchors.csv"); if (file.is_open()) { std::string line; while (std::getline(file, line)) { std::istringstream ss(line); std::string token; std::vector anchor; while (std::getline(ss, token, ',')) { anchor.push_back(std::stof(token)); } anchors.push_back(convertAnchorVectorToRect(anchor)); } file.close(); } return anchors; }

4. Change `NUM_BOXES_HAND` to 2944 instead of 2016
You can refer the above repo if you want to rotate the ROI. I hope this will help. Once again, sorry for make you wait for a really long time.
Thanks