Lib usage

[Vindicom]

Hello, one question about lib usage. I would like to use c# Bitmap (or raw image data array) as the way to communicate with your openpose wrapper (right now Mat is used but it is not clear how to convert Bitmat to Mat and vice versa). Do you maybe have an example code in c# showing how to do that?

[takuya-takeuchi]

No. There is no way to convert bitmap image to Mat class in openposedotnet for now. But op_wrapper_emplaceAndPop_rawImage is in OpenPoseDotNet.Native. It is experimental and I did not test yet.

So I am able to choice the following approach.

1. expose op_wrapper_emplaceAndPop_rawImage as wrapper method
2. Refer https://www.nuget.org/packages/System.Drawing.Common/4.5.1 and you can treat Bitmap directory.

But I am very busy and I won't work on immediately.

[Vindicom]

OK, I understand & I'll try to connect to op_wrapper_emplaceAndPop_rawImage (I see that it just creates Mat wrapper around the array which is good enough). One more question. Result of the algorithm (i.e. Mat object) will be placed in element with the name CvOutputData. I can see that Mat only have native pointer. Do you maybe know how can I come to the data array of the calculated image using this (Mat native) pointer?

[takuya-takeuchi]

Basically user needs not to consider NativePointer. So I conseal NativePointer property.

Therefore, it is better to expose methods to convert to byte[] or Bitmap in Mat class. Does it satisfy your requirements?

[Vindicom]

Adding conversion methods to your Mat would be great.

[Vindicom]

One more comment... it is probably easier to just export raw byte array. Then you can construct the Bitmap on the managed side. Assumption is that openpose will not change width, height or format of the bitmap that is given as input to op_wrapper_emplaceAndPop_rawImage. And just to clarify the use case... we are talking about enabling the streaming use case with your wrapper and openpose. Thanks!

[Vindicom]

Hello, While using op_wrapper_emplaceAndPop_rawImage() I've got exception shown below. I am pretty sure the array contains raw bytes of BGR with 3 bytes per pixel. I've used CV_8UC3 as type parameter. Input array does not contain any bitmap header, just bytes related to the pixels. Is this header needed? Do you maybe know what could be the problem?

Error: Input images must be 3-channel BGR.

Coming from:

• C:\Temp\OpenPoseDotNet-master\openpose\src\openpose\core\cvMatToOpInput.cpp:op::CvMatToOpInput::createArray():27
• C:\Temp\OpenPoseDotNet-master\openpose\src\openpose\core\cvMatToOpInput.cpp:op::CvMatToOpInput::createArray():55
• C:\Temp\OpenPoseDotNet-master\openpose\include\openpose/core/wCvMatToOpInput.hpp:op::WCvMatToOpInput<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > > >::work():80
• C:\Temp\OpenPoseDotNet-master\openpose\include\openpose/thread/subThread.hpp:op::SubThread<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > > > > >::workTWorkers():135
• C:\Temp\OpenPoseDotNet-master\openpose\include\openpose/thread/subThreadQueueInOut.hpp:op::SubThreadQueueInOut<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > > > >,class op::Queue<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > >,class std::queue<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > >,class std::deque<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > >,class std::allocator<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > > > > > > >::work():105
• C:\Temp\OpenPoseDotNet-master\openpose\include\openpose/thread/thread.hpp:op::Thread<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<struct op::Datum,class std::allocator > > > > >::threadFunction():203

[takuya-takeuchi]

How do you create raw image byte[]? If you create it from System.Drawing.Bitmap or BitmapSource, it contains padding. So it occurs problem.

Is this header needed

No

Do you maybe know what could be the problem?

This C++ wrapper creates cv::Mat instance and it is passed to openpose procee in function but it may be destruct even though it was passed to openpose. I guess this cv::Mat maybe should be cv::Mat*.

Could you show the your code to reproduce issue?

[Vindicom]

Size of bitmap array is exactly widthheight3 bytes so that looks ok... But I suspect that maybe my dll export is maybe done in wrong way... Can you please check?... Below are signatures of both versions of emaplaceAndPop:

[DllImport(NativeMethods.NativeLibrary, CallingConvention = NativeMethods.CallingConvention)] public static extern IntPtr op_wrapper_emplaceAndPop_cvMat(OpenPose.DataType dataType, IntPtr wrapper, IntPtr mat);

[DllImport(NativeMethods.NativeLibrary, CallingConvention = NativeMethods.CallingConvention)] public static extern IntPtr op_wrapper_emplaceAndPop_rawImage(OpenPose.DataType dataType, IntPtr wrapper, byte[] data, int width, int height, int type);

[takuya-takeuchi]

I tried Wrapper.EmplaceAndPop(byte[] image, int width, int height, int type).

        [DllImport(NativeMethods.NativeLibrary, CallingConvention = NativeMethods.CallingConvention)]
        public static extern IntPtr op_wrapper_emplaceAndPop_rawImage(OpenPose.DataType dataType,
                                                                      IntPtr wrapper,
                                                                      byte[] data,
                                                                      int width,
                                                                      int height,
                                                                      int type);

        public StdSharedPtr<StdVector<T>> EmplaceAndPop(byte[] image, int width, int height, int type) 
        { 
            if (image == null) 
                throw new ArgumentNullException(nameof(image)); 

            var ret = NativeMethods.op_wrapper_emplaceAndPop_rawImage(this._DataType, 
                                                                      this.NativePtr, 
                                                                      image, 
                                                                      width, 
                                                                      height, 
                                                                      type); 
            return new StdSharedPtr<StdVector<T>>(ret); 
        } 

It works fine. At first, I passed wrong value as int type. It produded same issue you faced. This time. type was 17. Correct value is 16. I guess your CV_8UC3 may be wrong value.

[takuya-takeuchi]

Could you test 334341e80757226107b7353201a379d87c1c6c69? 1_BodyFromImage_Bitmap example uses new EmplaceAndPop.

[Vindicom]

You are right, had a wrong value for CV_8UC3. Number 16 is correct. It is working for me also. By the way, just to confirm, IntPtr when passing array also works (on C# side you need to pin the array and get the pointer). E.g.

public StdSharedPtr<StdVector<T>> EmplaceAndPopRawImage(IntPtr data, int width, int height, int type) { if (data == null) throw new ArgumentNullException(nameof(data)); var ret = NativeMethods.op_wrapper_emplaceAndPop_rawImage(this._DataType, this.NativePtr, data, width, height, type); return new StdSharedPtr<StdVector<T>>(ret); }

[takuya-takeuchi]

You are right. But IntPtr is not used for VB.NET. I don't want to expose IntPtr signature as much as possible. And byte[] is Bittable type. So CLR marshaller make good. Sure, do pinning also. So need not to pin. You can refer https://docs.microsoft.com/ja-jp/dotnet/framework/interop/blittable-and-non-blittable-types

[Vindicom]

Yes, I agree byte[] is OK.

I am having problems now accessing Array values where pose key points are stored. There are two problems I see:

1. I can't get to the values using Array indexer i.e. [] operator.
2. Most of the times I can get to values using Array.toString() (and I see correct values), but sometimes this method goes into memory access violation exception. Before I access the array I do check that Array is existing and is not empty. Can you maybe look into why is 1. happening?

[takuya-takeuchi]

Hi. I added Mat,ToBitmap and Wrapper.EmplaceAndPop(Bitmap) to latest develop branch. Give it a try!!

I can't get to the values using Array indexer i.e. [] operator.

Do you have a image file which be able to reproduce issue? Or you faced issue while using webcam? If so, it is difficult to resolve issue.

[Vindicom]

OK, thanks, I will try it.

Do you have a image file which be able to reproduce issue?

The problem is not in the image... image is processed OK, but afterwards, calculated key points can't be accessed using indexer of Array< float >. By the way, I can get to calculated points using toString() function of Array< float >. It seems to me that indexer code is not correct.

[takuya-takeuchi]

Just in case, could you check the https://github.com/takuya-takeuchi/OpenPoseDotNet/blob/master/examples/TutorialApiCpp/8_SynchronousCustomOutput/WUserOutput.cs ? This sample works fine. And I have never seen problem you faced.

I doubt Array object is corrupted by GC.