Closed bruce987 closed 4 years ago
@bruce987 No. OpenPoseDotNet does not expose VideoCapture interface of opencv. We must export it into OpenPoseDotNet.Native and add p/invoke to OpenPoseDotNet.
I think you want really to create Mat object from OpenPoseDotNet. So
Sorry. We already exposed op_3rdparty_cv_Mat_data but Mat.data does not exist.
Thanks for the response. I will take a look at doing that.
Currently I have the image (OpenPose Mat) using this call…
using var opMat = OpenPose.OP_OP2CVCONSTMAT(temp[0].Get().CvOutputData);
I’ve tried taking the “opMat” object and converting it back to a OpenCv Mat object.
The conversion process does this…
var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);
That displays garbage, then after about 10 cycles throws an exception… “AccessViolationException”
-Bruce
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 4:41 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
@bruce987 https://github.com/bruce987 No. OpenPoseDotNet does not expose VideoCapture interface of opencv. We must export it into OpenPoseDotNet.Native and add p/invoke to OpenPoseDotNet.
I think you want really to create Mat object from OpenPoseDotNet. So
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622579582 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLY7FAB3M4EOD6CW273RPM6VFANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL5DIVZWV2EIPHWAN7TRPM6VFA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN467Q.gif
I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 4:46 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
Sorry. We already exposed op_3rdparty_cv_Mat_data but Mat.data does not exist.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622580924 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLYG5DFD6Y6KNLN5LATRPM7HZANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL5VVOGBPWWLPSNGO6TRPM7HZA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN5JPA.gif
I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.
openpose complete conseal videocapture interface, so I reluctant to expose it from OpenPoseDotNet.
var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);
What is image
?
NativePtr is not raw image data but image class object pointer.
I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.
openpose complete conseal videocapture interface, so I reluctant to expose it from OpenPoseDotNet.
Well one of the awesome demo’s that the OpenPose developers show is the live VideoCapture… I’ve demonstrated this at work and everyone was excited. We investigating this technology for NASA. I said there was a DotNet version. We are mostly code agnostic, but we are also a Microsoft partner. I prefer to work in DotNet C#. I’m appealing to you to expose that interface. I could possibly share my code and you could use it as a demo/example.
The method interface is…
private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image)
So image is the OpenPoseDotNet.Mat produced by the call to…
OpenPose.OP_OP2CVCONSTMAT(…)
var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);
What is image? NativePtr is not raw image data but image class object pointer.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622583138 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL54QCLVAFOPZEKAWLLRPNAFTANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBLZBXF6MN5P3VOCBP7DRPNAFTA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN52YQ.gif
Here’s some of my complete methods…
private static void Display(int rows, int cols, StdSharedPtr<StdVector<StdSharedPtr
{
try
{
// User's displaying/saving/other processing here
// datum.cvOutputData: rendered frame with pose or heatmaps
// datum.poseKeypoints: Array<float> with the estimated pose
if (datumsPtr != null && datumsPtr.TryGet(out var data) && !data.Empty)
{
// Display image
var temp = data.ToArray();
using var opMat = OpenPose.OP_OP2CVCONSTMAT(temp[0].Get().CvOutputData);
using var cvMat = CreateOpenCvMat(rows, cols, opMat);
Cv2.ImShow("Camera Output", cvMat);
}
else
{
LogHelper.Warn($"{MethodBase.GetCurrentMethod()}: Nullptr or empty datumsPtr found.");
}
}
catch (Exception e)
{
LogHelper.Error($"{MethodBase.GetCurrentMethod()}: {e.Message}");
}
}
This is the conversion method…
The “image.ToBitmap” throws an exception, so I’m not completely sure the OpenPose.OP2CVCONSTMAT is creating a complete Mat object.
private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image)
{
//var bitmap = image.ToBitmap();
//var bd = bitmap.LockBits(new Rectangle(0, 0, bitmap.Width, bitmap.Height),
// ImageLockMode.ReadWrite,
// PixelFormat.Format24bppRgb);
//IntPtr pVal;
//try
//{
// pVal = bd.Scan0;
//}
//finally
//{
// bitmap.UnlockBits(bd);
//}
//var bitmap = image.ToBitmap();
//var arr = bitmap.ToByteArray(ImageFormat.Bmp);
var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);
return cvMat;
}
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 4:53 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
I was hoping the Producer code would have the VideoCapture interface, but It doesn’t look like its complete.
openpose complete conseal videocapture interface, so I reluctant to expose it from OpenPoseDotNet.
var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, image.NativePtr);
What is image? NativePtr is not raw image data but image class object pointer.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622583138 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL54QCLVAFOPZEKAWLLRPNAFTANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBLZBXF6MN5P3VOCBP7DRPNAFTA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUN52YQ.gif
Sorry. I'm not familiar with English. In conclusion, you want me to do the following, right?
private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image)
{
var data = image.Data; // Add new interface
var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, data);
return cvMat;
}
That’s ok about the English…
I would prefer to have VideoCapture, then I can eliminate all of the conversions from OpenCvSharp and OpenPoseDotNet. This would provide a similar functionality like the CMU-Perceptual-Computing-Lab/openpose version.
In the meantime if doing #2 could be done quicker and provide a more robust API, that would be great.
-Bruce
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 5:44 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
Sorry. I'm not familiar with English. In conclusion, you want me to do the following, right?
private static OpenCvSharp.Mat CreateOpenCvMat(int rows, int cols, OpenPoseDotNet.Mat image) { var data = image.Data; // Add new interface var cvMat = new OpenCvSharp.Mat(rows, cols, MatType.CV_8UC3, data); return cvMat; }
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622597791 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL5IU6ILDUDGHNZPU5LRPNGB5ANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL5C44W7KQT42CVK7ODRPNGB5A5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOBNHY.gif
It may seem like I'm repeating myself. Does this code not satisfy your requirements?
Flags,Video can accept video file path. Flags.Camera is vidoe index and it is similar with VideoCapture. Flags.IpCamera is address of network camera (HTTP or RSTP)
var tie = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);
Or you can use OpenPoseDotNet.Cv.ImShow.
Anyway, you look like to need not converting to OpenCV.Mat for only showing image as Window.
That’s ok, it’s hard to find all the different API’s for a library that I’m not familiar with.
I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?
I will try the Cv.ImShow, but that didn’t seem to work, because I was using the OpenCvSharp library. This is where I was having to convert the different ‘Mat” objects. If I can hook up my webcam with OpenPoseDotNet that will solve the problems. I’m in a “while” loop capturing images and trying to outline each person in the frame.
-Bruce
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:11 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
It may seem like I'm repeating myself. Does this code not satisfy your requirements?
Flags,Video can accept video file path. Flags.Camera is vidoe index and it is similar with VideoCapture. Flags.IpCamera is address of network camera (HTTP or RSTP)
var tie = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);
Or you can use OpenPoseDotNet.Cv.ImShow.
Anyway, you look like to need not converting to OpenCV.Mat for only showing image as Window.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622605499 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL4DZXT7BJ5SMBFIEO3RPNJHLANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL4AU3BPKV7VTUQAJKTRPNJHLA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUODJOY.gif
I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?
Sorry. typo. RSTP is RTSP.
For WebCam, you should use Flags.Camera rather than Flags.IpCamera. IpCamera means network camera which requires LAN cable.
Flags.Camera = -1; // The camera index for cv::VideoCapture. Integer in the range [0, 9]. Select a negative
// number (by default), to auto-detect and open the first available camera.
For the var tie = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);
I’m not using ImageDir, IpCamera, FlirCamera, FlirCameraIndex. What do I do for those flags?
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:26 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?
Sorry. typo. RSTP is RTSP.
For WebCam, you should use Flags.Camera rather than Flags.IpCamera. IpCamera means network camera which requires LAN cable.
Flags.Camera = -1; // The camera index for cv::VideoCapture. Integer in the range [0, 9]. Select a negative
// number (by default), to auto-detect and open the first available camera.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622609423 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLYX55NHL6AEPOTZ4R3RPNLBFANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL7GAXBEJTFKXNZKF6DRPNLBFA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOEIDY.gif
My CameraCapture looks like this
private static void StartCameraCapture()
{
var preferredCamera = ConfigurationManager.AppSettings["preferredCamera"];
var cameraList = GetCameraList();
var cameraIndex = GetPreferredCamera(cameraList, preferredCamera.ToLowerInvariant());
if (cameraIndex != -1)
{
This is the suggestions you made. I see the FlagsToProducer returns a tuple of 2 items.
Flags.Video = preferredCamera;
Flags.Camera = cameraIndex;
var tie = OpenPose.FlagsToProducer(String.Empty, Flags.Video, String.Empty, Flags.Camera, false, 0);
What is the equivalent to the following?
var capture = new VideoCapture(cameraIndex);
capture.Open(cameraIndex);
if (capture.IsOpened())
{
ProcessImage(capture);
}
else
{
throw new Exception("Camera captured failed!");
}
}
else
{
throw new Exception($"Unable to find requested camera... {preferredCamera}!");
}
}
private static void ProcessImage(VideoCapture capture)
{
opWrapper.Start();
using var image = new Mat();
while (true)
{
What is the equivalent of the following?
This gets the video image put into the Mat.
capture.Read(image);
if (image.Empty())
{
break;
}
ParseCameraFrame(opWrapper, image);
// hit any key to exit, the 'X' doesn't work
var c = Cv2.WaitKey(1);
if (c != -1)
{
break;
}
}
}
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:26 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
I can try that. For Flags.IpCamera I’m using a WebCam, not sure what RSTP means?
Sorry. typo. RSTP is RTSP.
For WebCam, you should use Flags.Camera rather than Flags.IpCamera. IpCamera means network camera which requires LAN cable.
Flags.Camera = -1; // The camera index for cv::VideoCapture. Integer in the range [0, 9]. Select a negative
// number (by default), to auto-detect and open the first available camera.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622609423 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBLYX55NHL6AEPOTZ4R3RPNLBFANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL7GAXBEJTFKXNZKF6DRPNLBFA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOEIDY.gif
You can refer examples\TutorialApiCpp\14_SynchronousCustomPreProcessing and examples\TutorialApiCpp\15_SynchronousCustomPostProcessing. It could use WebCam and display modiied camera image before/after processing.
Ok I’m looking at the code. I will try and run it to see what is going on.
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:46 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
You can refer examples\TutorialApiCpp\14_SynchronousCustomPreProcessing and examples\TutorialApiCpp\15_SynchronousCustomPostProcessing. It could use WebCam and display modiied camera image before/after processing.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622613133 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL5EESIZZHMZWBSVFE3RPNNJVANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL67CCWTAQDHF7U3WZ3RPNNJVA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOFFDI.gif
Alright I got it running (had to modify the SymlinkBinary.ps1)!!!
Thanks for all the help.
Bruce
From: Takuya Takeuchi notifications@github.com Sent: Friday, May 1, 2020 6:46 PM To: takuya-takeuchi/OpenPoseDotNet OpenPoseDotNet@noreply.github.com Cc: bruce987 bruce.w.hochstetler@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuya-takeuchi/OpenPoseDotNet] OpenPoseDotNet not playing well with OpenCvSharp4 (#14)
You can refer examples\TutorialApiCpp\14_SynchronousCustomPreProcessing and examples\TutorialApiCpp\15_SynchronousCustomPostProcessing. It could use WebCam and display modiied camera image before/after processing.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuya-takeuchi/OpenPoseDotNet/issues/14#issuecomment-622613133 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCBL5EESIZZHMZWBSVFE3RPNNJVANCNFSM4MWAVW3Q . https://github.com/notifications/beacon/AAMCBL67CCWTAQDHF7U3WZ3RPNNJVA5CNFSM4MWAVW32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUOFFDI.gif
Hi Takuya,
I have an application that uses OpenCvSharp4. I use it for its camera video capture. I'm using your OpenPoseDotNet to essential do the same thing as the OpenPose demo does... capture video and outline person(s).
The problem is OpenCvSharp4 generates a OpenCvSharp.Mat object. The OpenPoseDotNet is looking for an OpenPoseDotNet.Mat object. I've attempted to convert the OpenCvSharp.Mat by converting it to a bitmap then back to an OpenPoseDotNet.Mat. The call OpenPoseDotNet. OP_OP2CVCONSTMAT in the Display method (taken from the example), supposedly creates a OpenPoseDotNetMat.Mat. To use the Cv2.ImShow method requires me to convert the OpenPoseDotNetMat.Mat back to a OpenCvSharp.Mat. It fails trying to call the ToBitmap method.
I tried to locate a VideoCapture method in the OpenPoseDotNet library, but couldn't locate it. I would much rather work with a single Mat object. If you could point me to the class that handles VideoCapture with maybe an example that would be most helpful.
Thanks, Bruce