search

ThorstenBux / artoolkitX_em_2d

2d tracking with emscripten
Other
0 stars 1 forks source link

WIP - converting to OpenCL - 1 test #3

Open kalwalt opened 5 years ago

kalwalt commented 5 years ago

i'm testing the conversion to OpenCL: basically converting the part of the code where OpenCL can improve the performances, it should only necessary change some cv::Mat to cv::UMat. This PR is most a trial and error, i will report in the comments the problems that may arise.

kalwalt commented 5 years ago

Whit the latest i get this error:

/home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/PlanarTracker.cpp:415:20: error: 
      no matching constructor for initialization of 'cv::UMat'
  ...colorFrame(_frameSizeY, _frameSizeX, CV_8UC4, frame, cv::USAGE_DEFAULT);
     ^          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/walter/kalwalt-github/artoolkitX_em_2d/Source/depends/emscripten/opencv-4.1/modules/core/include/opencv2/core/mat.inl.hpp:3600:7: note: 
      candidate constructor not viable: no known conversion from
      'unsigned char *' to 'const cv::Scalar' (aka 'const Scalar_<double>') for
      4th argument
UMat::UMat(int _rows, int _cols, int _type, const Scalar& _s, UMatUsageF...
      ^

cv::Mat instead may accept void* data 

Mat (int rows, int cols, int type, void *data, size_t step=AUTO_STEP)

from the OpenCV docs

kalwalt commented 5 years ago

I read this article https://www.learnopencv.com/opencv-transparent-api/ and i solved sending the data initially to a cv::Mat and after getting them with the getUmat function. Any way this is fixed only for the Emscripten part, won't work for the other platforms.

  void ProcessFrameData(unsigned char * frame)
    {
      cv::Mat data(_frameSizeY, _frameSizeX, CV_8UC4, frame);
      // When using emscripten the image comes in as RGB image from the browser
      // Convert it to Gray
      #if ARX_TARGET_PLATFORM_EMSCRIPTEN
      cv::UMat colorFrame = data.getUMat(cv::ACCESS_READ);
kalwalt commented 5 years ago

This error with the latest commit:

/home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/PlanarTracker.cpp:724:52: error: 
      no member named 'ptr' in 'cv::UMat'
                memcpy(data, _trackables[i]._image.ptr(), _trackables[i]...
                             ~~~~~~~~~~~~~~~~~~~~~ ^
In file included from /home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/PlanarTracker.cpp:43:
In file included from /home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/OCVConfig.h:43:
In file included from /home/walter/kalwalt-github/artoolkitX_em_2d/Source/depends/emscripten/opencv-4.1/modules/core/include/opencv2/core.hpp:60:

Needs to think how to solve...

kalwalt commented 5 years ago

@ThorstenBux I have my Artoolkitx.js and Artoolkitx.wasm ready!

kalwalt commented 5 years ago

The code do not detect the marker but i get an decisive increase in fps, probably the reason is in the changes in passing the data. If you want to try i let here the compiled libs. artoolkitxCL.zip

kalwalt commented 5 years ago

sure the issue is in the PlanarTracker.cpp 

TrackedImageInfo GetTrackableImageInfo(int trackableId)
    {
        TrackedImageInfo info;
        for(int i=0;i<_trackables.size(); i++) {
            if(_trackables[i]._id==trackableId) {
                info.uid = _trackables[i]._id;
                info.scale = _trackables[i]._scale;
                info.fileName = _trackables[i]._fileName;
                // Copy the image data and use a shared_ptr to refer to it.
                unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height);
                /// this needs to be verifyed!!
                cv::Mat dataPtr(_trackables[i]._width, _trackables[i]._height, CV_8UC4, data);
                _trackables[i]._image = dataPtr.getUMat(cv::ACCESS_READ);
                //memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height);
                info.imageData.reset(data, free);
                info.width = _trackables[i]._width;
                info.height = _trackables[i]._height;
                info.fileName = _trackables[i]._fileName;
                return info;
            }
        }
        return info;
    }

before was:

TrackedImageInfo GetTrackableImageInfo(int trackableId)
    {
        TrackedImageInfo info;
        for(int i=0;i<_trackables.size(); i++) {
            if(_trackables[i]._id==trackableId) {
                info.uid = _trackables[i]._id;
                info.scale = _trackables[i]._scale;
                info.fileName = _trackables[i]._fileName;
                // Copy the image data and use a shared_ptr to refer to it.
                unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height);
                memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height);
                info.imageData.reset(data, free);
                info.width = _trackables[i]._width;
                info.height = _trackables[i]._height;
                info.fileName = _trackables[i]._fileName;
                return info;
            }
        }
        return info;
    }

the problem is we can not have a ptr() for the UMat, that Mat instead has. If we fix this maybe the code will works correctly. Proabably we can do other optimizations with Umat as for example in CameraPoseFromPoints and some other part of the code too.

ThorstenBux commented 5 years ago

That is amazing work!

I had done some UMat things a way back. Will have a look at what you did. It might probably be next week though, we are planning to head out for a long weekend and there is a lot todo this week.

Get Outlook for iOShttps://aka.ms/o0ukef

From: Walter Perdan notifications@github.com Sent: Tuesday, September 3, 2019 6:16:24 AM To: ThorstenBux/artoolkitX_em_2d artoolkitX_em_2d@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [ThorstenBux/artoolkitX_em_2d] WIP - converting to OpenCL - 1 test (#3)

sure the issue is in the PlanarTracker.cpp

TrackedImageInfo GetTrackableImageInfo(int trackableId) { TrackedImageInfo info; for(int i=0;i<_trackables.size(); i++) { if(_trackables[i]._id==trackableId) { info.uid = _trackables[i]._id; info.scale = _trackables[i]._scale; info.fileName = _trackables[i]._fileName; // Copy the image data and use a shared_ptr to refer to it. unsigned char data = (unsigned char )malloc(_trackables[i]._width _trackables[i]._height); /// this needs to be verifyed!! cv::Mat dataPtr(_trackables[i]._width, _trackables[i]._height, CV_8UC4, data); _trackables[i]._image = dataPtr.getUMat(cv::ACCESS_READ); //memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width _trackables[i]._height); info.imageData.reset(data, free); info.width = _trackables[i]._width; info.height = _trackables[i]._height; info.fileName = _trackables[i]._fileName; return info; } } return info; }

before was:

TrackedImageInfo GetTrackableImageInfo(int trackableId) { TrackedImageInfo info; for(int i=0;i<_trackables.size(); i++) { if(_trackables[i]._id==trackableId) { info.uid = _trackables[i]._id; info.scale = _trackables[i]._scale; info.fileName = _trackables[i]._fileName; // Copy the image data and use a shared_ptr to refer to it. unsigned char data = (unsigned char )malloc(_trackables[i]._width _trackables[i]._height); memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width _trackables[i]._height); info.imageData.reset(data, free); info.width = _trackables[i]._width; info.height = _trackables[i]._height; info.fileName = _trackables[i]._fileName; return info; } } return info; }

the problem is we can not have a ptr() for the UMat, that Mat instead has. If we fix this maybe the code will works correctly. Proabably we can do other optimizations with Umat as for example in CameraPoseFromPoints and some other part of the code too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux%2FartoolkitX_em_2d%2Fpull%2F3%3Femail_source%3Dnotifications%26email_token%3DAD765PGMUQ5TCMCJVE46UFTQHVJ7RA5CNFSM4IS6DJK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5WLK6A%23issuecomment-527218040&data=02%7C01%7C%7C6605fd082de941e1156708d72fd1aa7f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637030449856984036&sdata=vnxImNDWLPdjgE1FQZADlqmYOjkOfA8AfXdRNVPcecw%3D&reserved=0, or mute the threadhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PCYZCRAB65WSIU2K2TQHVJ7RANCNFSM4IS6DJKQ&data=02%7C01%7C%7C6605fd082de941e1156708d72fd1aa7f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637030449856984036&sdata=4%2F7TYhaW7MoTP8GXqnrMrQQcUQzxnir31p0qu9QT398%3D&reserved=0.

kalwalt commented 5 years ago

Yes, but i was very lucky too! The Transparent Api helps a lot! We will see how to solve, no problem when. Good holiday!

ThorstenBux commented 5 years ago

I've pushed my branch of UMat things that I had done previously: https://github.com/ThorstenBux/artoolkitX_em_2d/tree/perf_OpenCL_UMat

You can have a look at them if you like :).

kalwalt commented 5 years ago

Ok thank you very much! For sure i will look at!

ThorstenBux commented 5 years ago

Did you compare to my umat changes? I thought they did work at some stage

Get Outlook for iOShttps://aka.ms/o0ukef

From: Walter Perdan notifications@github.com Sent: Wednesday, September 4, 2019 6:53:48 PM To: ThorstenBux/artoolkitX_em_2d artoolkitX_em_2d@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [ThorstenBux/artoolkitX_em_2d] WIP - converting to OpenCL - 1 test (#3)

@kalwalt commented on this pull request.

In Source/ARX/OCVT/PlanarTracker.cpphttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux%2FartoolkitX_em_2d%2Fpull%2F3%23discussion_r320598028&data=02%7C01%7C%7C673a3bd706ad406b458808d73104a3d7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637031768301731204&sdata=%2FgjA6ic9oUWMzEzhRM4sXn15t2gljambsJTb943DIfM%3D&reserved=0:

       cv::UMat grayImage(_frameSizeY, _frameSizeX, CV_8UC1);
  • cv::cvtColor(colorImage, grayImage, cv::COLOR_RGBA2GRAY);
  • std::cout << "grayImage ok!" << std::endl;

It refers to 'newTrackable' i think the jpg image. It seems strange to me that fails at this point. Unlucky i will not have time to try other changes...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux%2FartoolkitX_em_2d%2Fpull%2F3%3Femail_source%3Dnotifications%26email_token%3DAD765PHPK7OXOVLTGXNHGQ3QH5LPZA5CNFSM4IS6DJK2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCDSHRNY%23discussion_r320598028&data=02%7C01%7C%7C673a3bd706ad406b458808d73104a3d7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637031768301741222&sdata=4HrBB7iPWGEICfzdTcbc5gI4sH%2FvlqzpMUiFV2XU5Tg%3D&reserved=0, or mute the threadhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PHO727FUULWYDALCHDQH5LPZANCNFSM4IS6DJKQ&data=02%7C01%7C%7C673a3bd706ad406b458808d73104a3d7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637031768301761232&sdata=5TzK1GtdgiZFo44sEtobutyKZKx5mFnLmty3efUWonk%3D&reserved=0.

kalwalt commented 5 years ago

i did but very quickly and not for all the code, i should do indeed !

kalwalt commented 5 years ago

The code actually seems to fail when loading the data image to be tracked (in the addMarker function) not understand why it can not loaded, i don't see such critical code in addMarker or maybe it fails silently in another part but i can not get where.

ThorstenBux commented 5 years ago

Hi,

I start to feel that OpenCL won't work on mobile. Some things that I'm reading point into the direction that the drivers need to be written for the specific GPU and things. Which might explain why it starts failing. Basically, the browser doesn't have access to it but we tell the transpiled OpenCV that OpenCL is available so it tries to call it but fails because it isn't really there. The web counterpart would be WebCL which in turn isn't implemented inside any browser yet and the Emscripten polyfill is reportedly slower than not using it at all.

We might need to find another solution to tweak the performance

kalwalt commented 5 years ago

yes i can understand this. But could be an issue the fact i build the OpenCV lib with "-DWITH_OPENCL=ON -DWITH_OPENCL_SVM=ON" with OpenCL enabled but also with OpenCL and the Shared Virtual Memory enabled? I can try to test without this last one, just to understand if it make a difference.

ThorstenBux commented 5 years ago

Already tried that without success.

ThorstenBux commented 5 years ago

We would need to find some other idea to improve performance. I fear

kalwalt commented 5 years ago

ok good to know, and basically have you some ideas for other improvements?

ThorstenBux commented 5 years ago

Well, first would be to get it running profile it to see where the bottleneck is.

One idea would be to run a rectangle detection over the image first to reduce the size of the image that needs to run the recognition and tracking.

ThorstenBux commented 5 years ago

I see you are working with https://github.com/kalwalt/MarkerlessARJS maybe they did something different which could improve performance?

I recall that the performance might be solvePnP but not entirely sure. I'd say let us aim for a profiling run? Could you do that?

kalwalt commented 5 years ago

Well, first would be to get it running profile it to see where the bottleneck is.

yes you are right, this is important.

One idea would be to run a rectangle detection over the image first to reduce the size of the image that needs to run the recognition and tracking.

Do you mean a rectangle detection thanks to OpenCV, like a blob detection algorithm?

I see you are working with https://github.com/kalwalt/MarkerlessARJS maybe they did something different which could improve performance?

yes i'm working on this but i'am far to make something that can be tested, The original code was developed with a oldest version of OpenCV. I hope that it can be something better in performances, i hope to go on tihs to compare with ArtoolkitX.js, The original code was only for windows I think, didn't tested the original one. The author says also in this issue https://github.com/ahmetozlu/augmented_reality/issues/1 about Dimensionality Reduction maybe could be an idea?

I recall that the performance might be solvePnP but not entirely sure. I'd say let us aim for a profiling run? Could you do that?

I will try it i f i have time tomorrow or in the next days. Actually i am in France for job, i arrived today.

kalwalt commented 5 years ago

also read this issue: https://github.com/ahmetozlu/augmented_reality/issues/2

ThorstenBux commented 5 years ago

also read this issue: ahmetozlu/augmented_reality#2

Detection and tracking is already separated inside the C function as far as I know but it is not running as two threads that would be an idea.

kalwalt commented 5 years ago

Detection and tracking is already separated inside the C function as far as I know but it is not running as two threads that would be an idea.

Can we do seperate threads with Emscripten? we can but enabling -s USE_PTHREADS=1 right? as i did for jsartoolkit5 in https://github.com/kalwalt/jsartoolkit5/pull/2/ but it is needed to enable the shared memory option on the phone. Or maybe you mean something else.

ThorstenBux commented 5 years ago

I've compiled with PTHREADS before but didn't know about the shared memory option?

kalwalt commented 5 years ago

I've compiled with PTHREADS before but didn't know about the shared memory option?

yes it is. maybe you used before chrome and other browsers disabled the default. But maybe in a near future it will become a default option again. I will try to do the profiling tomorrow.

kalwalt commented 5 years ago

i m doing the profiling i will open a separate issue for this.