search

takuya-takeuchi / OpenPoseDotNet

OpenPose wrapper written in C++ and C# for Windows
MIT License
66 stars 25 forks source link

OpenPoseDotNet not playing well with OpenCvSharp4 #14

Closed bruce987 closed 4 years ago

bruce987 commented 4 years ago

Hi Takuya,

I have an application that uses OpenCvSharp4. I use it for its camera video capture. I'm using your OpenPoseDotNet to essential do the same thing as the OpenPose demo does... capture video and outline person(s).

The problem is OpenCvSharp4 generates a OpenCvSharp.Mat object. The OpenPoseDotNet is looking for an OpenPoseDotNet.Mat object. I've attempted to convert the OpenCvSharp.Mat by converting it to a bitmap then back to an OpenPoseDotNet.Mat. The call OpenPoseDotNet. OP_OP2CVCONSTMAT in the Display method (taken from the example), supposedly creates a OpenPoseDotNetMat.Mat. To use the Cv2.ImShow method requires me to convert the OpenPoseDotNetMat.Mat back to a OpenCvSharp.Mat. It fails trying to call the ToBitmap method.

I tried to locate a VideoCapture method in the OpenPoseDotNet library, but couldn't locate it. I would much rather work with a single Mat object. If you could point me to the class that handles VideoCapture with maybe an example that would be most helpful.

Thanks, Bruce

takuya-takeuchi commented 4 years ago

@bruce987 No. OpenPoseDotNet does not expose VideoCapture interface of opencv. We must export it into OpenPoseDotNet.Native and add p/invoke to OpenPoseDotNet.

I think you want really to create Mat object from OpenPoseDotNet. So

  1. expose opencv::Mat::data method and add p/invoke to OpenPoseDotNet.
  2. You read mat data from OpenPoseDotNet.Mat.data and write it to OpenCV4.Mat.
takuya-takeuchi commented 4 years ago

Sorry. We already exposed op_3rdparty_cv_Mat_data but Mat.data does not exist.

bruce987 commented 4 years ago

Thanks for the response. I will take a look at doing that.

Currently I have the image (OpenPose Mat) using this call…

using var opMat = OpenPose.OP_OP2CVCONSTMAT(temp[0].Get().CvOutputData);

I’ve tried taking the “opMat” object and converting it back to a OpenCv Mat object.

The conversion process does this…

var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);

That displays garbage, then after about 10 cycles throws an exception… “AccessViolationException”

-Bruce

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 4:41 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

@bruce987 https://github.com/bruce987 No. OpenPoseDotNet does not expose VideoCapture interface of opencv. We must export it into OpenPoseDotNet.Native and add p/invoke to OpenPoseDotNet.

I think you want really to create Mat object from OpenPoseDotNet. So

  1. expose opencv::Mat::data method and add p/invoke to OpenPoseDotNet.
  2. You read mat data from OpenPoseDotNet.Mat.data and write it to OpenCV4.Mat.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622579582 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLY7FAB3M4EOD6CW273RPM6VFANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL5DIVZWV2EIPHWAN7TRPM6VFA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN467Q.gif

bruce987 commented 4 years ago

I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 4:46 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

Sorry. We already exposed op_3rdparty_cv_Mat_data but Mat.data does not exist.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622580924 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLYG5DFD6Y6KNLN5LATRPM7HZANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL5VVOGBPWWLPSNGO6TRPM7HZA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN5JPA.gif

takuya-takeuchi commented 4 years ago

I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.

openpose complete conseal videocapture interface, so I reluctant to expose it from OpenPoseDotNet.

var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);

What is image? NativePtr is not raw image data but image class object pointer.

bruce987 commented 4 years ago

I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.

openpose complete conseal videocapture interface, so I reluctant to expose it from OpenPoseDotNet.

Well one of the awesome demo’s that the OpenPose developers show is the live VideoCapture… I’ve demonstrated this at work and everyone was excited. We investigating this technology for NASA. I said there was a DotNet version. We are mostly code agnostic, but we are also a Microsoft partner. I prefer to work in DotNet C#. I’m appealing to you to expose that interface. I could possibly share my code and you could use it as a demo/example.

The method interface is…

private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image)

So image is the OpenPoseDotNet.Mat produced by the call to…

OpenPose.OP_OP2CVCONSTMAT(…)

var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);

What is image? NativePtr is not raw image data but image class object pointer.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622583138 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL54QCLVAFOPZEKAWLLRPNAFTANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBLZBXF6MN5P3VOCBP7DRPNAFTA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN52YQ.gif

bruce987 commented 4 years ago

Here’s some of my complete methods…

private static void Display(int rows, int cols, StdSharedPtr<StdVector<StdSharedPtr>> datumsPtr)

    {

        try

        {

            // User's displaying/saving/other processing here

            // datum.cvOutputData: rendered frame with pose or heatmaps

            // datum.poseKeypoints: Array<float> with the estimated pose

            if (datumsPtr != null && datumsPtr.TryGet(out var data) && !data.Empty)

            {

                // Display image

                var temp = data.ToArray();

                using var opMat = OpenPose.OP_OP2CVCONSTMAT(temp[0].Get().CvOutputData);

                using var cvMat = CreateOpenCvMat(rows, cols, opMat);

                Cv2.ImShow("Camera Output", cvMat);

            }

            else

            {

                LogHelper.Warn($"{MethodBase.GetCurrentMethod()}: Nullptr or empty datumsPtr found.");

            }

        }

        catch (Exception e)

        {

            LogHelper.Error($"{MethodBase.GetCurrentMethod()}: {e.Message}");

        }

    }

This is the conversion method…

The “image.ToBitmap” throws an exception, so I’m not completely sure the OpenPose.OP2CVCONSTMAT is creating a complete Mat object.

private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image)

    {

        //var bitmap = image.ToBitmap();

        //var bd = bitmap.LockBits(new Rectangle(0, 0, bitmap.Width, bitmap.Height),

        //    ImageLockMode.ReadWrite,

        //    PixelFormat.Format24bppRgb);

        //IntPtr pVal;

        //try

        //{

        //    pVal = bd.Scan0;

        //}

        //finally

        //{

        //    bitmap.UnlockBits(bd);

        //}

        //var bitmap = image.ToBitmap();

        //var arr = bitmap.ToByteArray(ImageFormat.Bmp);

        var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);

        return cvMat;

    }

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 4:53 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.

openpose complete conseal videocapture interface, so I reluctant to expose it from OpenPoseDotNet.

var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);

What is image? NativePtr is not raw image data but image class object pointer.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622583138 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL54QCLVAFOPZEKAWLLRPNAFTANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBLZBXF6MN5P3VOCBP7DRPNAFTA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN52YQ.gif

takuya-takeuchi commented 4 years ago

Sorry. I'm not familiar with English. In conclusion, you want me to do the following, right?

  1. Add VideoCapture interface
  2. Resove conversion issue from OPDN.Mat to OpenCVSharp.Mat
    • I think I should only expose Data property.
private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image)
{
    var data = image.Data; // Add new interface
    var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, data);
    return cvMat;
}
bruce987 commented 4 years ago

That’s ok about the English…

I would prefer to have VideoCapture, then I can eliminate all of the conversions from OpenCvSharp and OpenPoseDotNet. This would provide a similar functionality like the CMU-Perceptual-Computing-Lab/openpose version.

In the meantime if doing #2 could be done quicker and provide a more robust API, that would be great.

-Bruce

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 5:44 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

Sorry. I'm not familiar with English. In conclusion, you want me to do the following, right?

  1. Add VideoCapture interface
  2. Resove conversion issue from OPDN.Mat to OpenCVSharp.Mat

private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image) { var data = image.Data; // Add new interface var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, data); return cvMat; }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622597791 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL5IU6ILDUDGHNZPU5LRPNGB5ANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL5C44W7KQT42CVK7ODRPNGB5A5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOBNHY.gif

takuya-takeuchi commented 4 years ago

It may seem like I'm repeating myself. Does this code not satisfy your requirements?

Flags,Video can accept video file path. Flags.Camera is vidoe index and it is similar with VideoCapture. Flags.IpCamera is address of network camera (HTTP or RSTP)

var tie = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);

Or you can use OpenPoseDotNet.Cv.ImShow.

Anyway, you look like to need not converting to OpenCV.Mat for only showing image as Window.

bruce987 commented 4 years ago

That’s ok, it’s hard to find all the different API’s for a library that I’m not familiar with.

I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?

I will try the Cv.ImShow, but that didn’t seem to work, because I was using the OpenCvSharp library. This is where I was having to convert the different ‘Mat” objects. If I can hook up my webcam with OpenPoseDotNet that will solve the problems. I’m in a “while” loop capturing images and trying to outline each person in the frame.

-Bruce

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:11 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

It may seem like I'm repeating myself. Does this code not satisfy your requirements?

Flags,Video can accept video file path. Flags.Camera is vidoe index and it is similar with VideoCapture. Flags.IpCamera is address of network camera (HTTP or RSTP)

var tie = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);

Or you can use OpenPoseDotNet.Cv.ImShow.

Anyway, you look like to need not converting to OpenCV.Mat for only showing image as Window.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622605499 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL4DZXT7BJ5SMBFIEO3RPNJHLANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL4AU3BPKV7VTUQAJKTRPNJHLA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUODJOY.gif

takuya-takeuchi commented 4 years ago

I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?

Sorry. typo. RSTP is RTSP.

For WebCam, you should use Flags.Camera rather than Flags.IpCamera. IpCamera means network camera which requires LAN cable.

Flags.Camera = -1; // The camera index for cv::VideoCapture. Integer in the range [0, 9]. Select a negative
                   // number (by default), to auto-detect and open the first available camera.
bruce987 commented 4 years ago

For the var tie = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);

I’m not using ImageDir, IpCamera, FlirCamera, FlirCameraIndex. What do I do for those flags?

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:26 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?

Sorry. typo. RSTP is RTSP.

For WebCam, you should use Flags.Camera rather than Flags.IpCamera. IpCamera means network camera which requires LAN cable.

Flags.Camera = -1; // The camera index for cv::VideoCapture. Integer in the range [0, 9]. Select a negative

               // number (by default), to auto-detect and open the first available camera.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622609423 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLYX55NHL6AEPOTZ4R3RPNLBFANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL7GAXBEJTFKXNZKF6DRPNLBFA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOEIDY.gif

bruce987 commented 4 years ago

My CameraCapture looks like this

private static void StartCameraCapture()

    {

        var preferredCamera = ConfigurationManager.AppSettings["preferredCamera"];

        var cameraList = GetCameraList();

        var cameraIndex = GetPreferredCamera(cameraList, preferredCamera.ToLowerInvariant());

        if (cameraIndex != -1)

        {

This is the suggestions you made. I see the FlagsToProducer returns a tuple of 2 items.

            Flags.Video = preferredCamera;

            Flags.Camera = cameraIndex;

            var tie = OpenPose.FlagsToProducer(String.Empty, Flags.Video, String.Empty, Flags.Camera, false, 0);

What is the equivalent to the following? 

            var capture = new VideoCapture(cameraIndex);

            capture.Open(cameraIndex);

            if (capture.IsOpened())

            {

                ProcessImage(capture);

            }

            else

            {

                throw new Exception("Camera captured failed!");

            }

        }

        else

        {

            throw new Exception($"Unable to find requested camera... {preferredCamera}!");

        }

    }

   private static void ProcessImage(VideoCapture capture)

    {                

        opWrapper.Start();

        using var image = new Mat();

        while (true)

        {

What is the equivalent of the following?

This gets the video image put into the Mat.

            capture.Read(image);

            if (image.Empty())

            {

                break;

            }

            ParseCameraFrame(opWrapper, image);

            // hit any key to exit, the 'X' doesn't work

            var c = Cv2.WaitKey(1);

            if (c != -1)

            {

                break;

            }

        }

    }

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:26 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?

Sorry. typo. RSTP is RTSP.

For WebCam, you should use Flags.Camera rather than Flags.IpCamera. IpCamera means network camera which requires LAN cable.

Flags.Camera = -1; // The camera index for cv::VideoCapture. Integer in the range [0, 9]. Select a negative

               // number (by default), to auto-detect and open the first available camera.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622609423 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLYX55NHL6AEPOTZ4R3RPNLBFANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL7GAXBEJTFKXNZKF6DRPNLBFA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOEIDY.gif

takuya-takeuchi commented 4 years ago

You can refer examples\TutorialApiCpp\14_SynchronousCustomPreProcessing and examples\TutorialApiCpp\15_SynchronousCustomPostProcessing. It could use WebCam and display modiied camera image before/after processing.

bruce987 commented 4 years ago

Ok I’m looking at the code. I will try and run it to see what is going on.

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:46 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

You can refer examples\TutorialApiCpp\14_SynchronousCustomPreProcessing and examples\TutorialApiCpp\15_SynchronousCustomPostProcessing. It could use WebCam and display modiied camera image before/after processing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622613133 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL5EESIZZHMZWBSVFE3RPNNJVANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL67CCWTAQDHF7U3WZ3RPNNJVA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOFFDI.gif

bruce987 commented 4 years ago

Alright I got it running (had to modify the SymlinkBinary.ps1)!!!

Thanks for all the help.

Bruce

From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:46 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)

You can refer examples\TutorialApiCpp\14_SynchronousCustomPreProcessing and examples\TutorialApiCpp\15_SynchronousCustomPostProcessing. It could use WebCam and display modiied camera image before/after processing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622613133 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL5EESIZZHMZWBSVFE3RPNNJVANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL67CCWTAQDHF7U3WZ3RPNNJVA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOFFDI.gif