Portrait orientation on Android

[opencv-pushbot]

Transferred from http://code.opencv.org/issues/3565

|| Sergey Abdula on 2014-02-20 04:06
|| Priority: Normal
|| Affected: 2.4.0 - 2.4.7
|| Category: android
|| Tracker: Bug
|| Difficulty: Easy
|| PR: 
|| Platform: ARM / Android

Portrait orientation on Android

Hi,
If to change  android:screenOrientation from landscape to portrait mode in "OpenCVTutorial 1 - Camera Preview", an image is rotated to 270 degree clockwise.

It's pretty basic feature for mobile applications (easy change orientation) and if I understand correctly, it was discussed last year (see http://stackoverflow.com/questions/14816166/rotate-camera-preview-to-portrait-android-opencv-camera). However, I could not find any opened or closed issue related to this problem.

There is kind of way to fix it (see http://littlecheesecake.me/blog/13804736/opencv-android-orientation) but it's required to rewrite the basic class (which is not good idea) and it does not work properly anyway.

Thanks in advance.

Sergey

History

Alexander Karsakov on 2014-02-20 06:53

-   Status changed from New to Open

Alexander Smorkalov on 2014-02-25 07:24

-   Target version set to 2.4.9
-   Difficulty set to Easy

Sergey Abdula on 2014-03-14 02:43

Hi Alex,

It will be nice to have a taken picture in correct orientation in portrait mode as well.

Alexander Smorkalov on 2014-04-01 23:12

We cannot change API behavior in 2.4 branch. I reasign the task on 3.0 release (master branch). It will not be compatible with 2.4.x, so we can change API behavour and fix such issues.
-   Target version changed from 2.4.9 to 3.0

Stephen G on 2015-05-06 02:09

Found the error in calculateCameraFrameSize in CameraBridgeViewBase.

A. mMaxWidth/Height are never set and should be (easily done by overriding onSizeChanged in the CameraView and calling setMaxFrameSize)

B. the supportedSizes list only list the max hieght/widths in one dimensions. the comparisons done in the weird double if statements need to be completed for width <= maxAllowedWidth ... *&& height <= maxAllowedWidth ...*. Then, the mScale calculate also needs to take this change into account with something like this: 
<pre>
mScale = Math.max(
    Math.min(((float) height) / mFrameHeight, ((float) width) / mFrameWidth),
    Math.min(((float) width) / mFrameHeight, ((float) height) / mFrameWidth)
);
</pre>

This solution doesn't fill the whole screen, but it does keep the proper ratio.

Does it work? could it be updated for 3 since it's still in an RC state?

Andrea Tomassetti on 2015-05-19 11:37

Alexander Smorkalov wrote:
> We cannot change API behavior in 2.4 branch. I reasign the task on 3.0 release (master branch). It will not be compatible with 2.4.x, so we can change API behavour and fix such issues.

Hi, I've just downloaded OpenCV 3.0RC1 for Android but there is the same issue. I'm wrong? It was fixed?

Thanks in advance

Fabian Hoyos on 2015-07-03 16:25

Hi, would I like to know if the bug Portrait orientation on Android is fixed in the version 3.0 of opencv Android SDK. I have the same problem when I use opencv Android sdk in mode portrait in the app to open the camera and face detection. 

Thanks for information. I hope soon know the solution, please.

[webknjaz]

@akarsakov any progress on this?

[antonov256]

Please, fix it! I'm waiting about 1,5 years.

[netcat88]

I think this might be related with what Stephen wrote, but I noticed the mScale should be the min between 2 ratio, one for height and one for width, while it always considers the height's one, even if width's is lower

[sbayd]

Is there any progress with this bug?

[Looney32768]

Capturing video from camera IMHO is a very common and basic need for almost any opencv project. On the other hand, rotating the device is also a common action every user is used to. And we've got a bug right at the point of intersection of these very common needs, and it's been around for more than 2.5 years already, with hack-ish and buggy workarounds scattered over the internet. That is absurd.

[michaelgobbers]

Like people mentioned here before. This is been an issue for way to long now... Is there at least an estimated timeframe for this fix?

[koala99]

this issue still exists in opencv 3.2. nobody solve ???

[zhenglaizhang]

still met with the issue in opencv 3.3 -:(

[ziarno]

@sergeychilingaryan where are you getting width, height, mFrameWidth and mFrameHeight in onPreviewFrame and onCreate from?

[ghost]

Hi all, i have the solution! 🗡

CHANGE

Math.min()

to --> Math.max()

Bug fixed noW!

Try to change this line in javaCameraView

FROM:

if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT))
                        mScale = Math.min(((float) height) / mFrameHeight, ((float) width) / mFrameWidth);                                      
                    else
                        mScale = 0;

TO:

if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT))
                        mScale = Math.max(((float) height) / mFrameHeight, ((float) width) / mFrameWidth);                                      
                    else
                        mScale = 0;

THIS "ONLY" MAKE FULLSCREEN THE FRAME, then flip the mat et voilà!

i'm using opencv 3_3_0 but i think it's the same in all versions.

C'mon that's was ez!!!

Solved in 2 hours :D love u

[jmgomezpoveda]

Roberto: you have posted the same code in the From and the To.

On 9 Oct 2017 09:49, "RobertoManfreda" notifications@github.com wrote:

Hi all, i have the solution! 🗡

Bug fixed noW!

Try to change this line in javaCameraView

FROM:

if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT)) mScale = Math.min(((float) height) / mFrameHeight, ((float) width) / mFrameWidth); else mScale = 0;

TO:

if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT)) mScale = Math.min(((float) height) / mFrameHeight, ((float) width) / mFrameWidth); else mScale = 0;

THIS "ONLY" MAKE FULLSCREEN THE FRAME, then flip the mat et voilà!

i'm using opencv 3_3_0 but i think it's the same in all versions.

C'mon that's was ez!!!

Solved in 2 hours :D love u

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/opencv/opencv/issues/4704#issuecomment-335098055, or mute the thread https://github.com/notifications/unsubscribe-auth/ADjNFL0sJpu4X1z1YNNub69u2AXpJZxcks5sqd4ygaJpZM4FgS7v .

[webknjaz]

@jmgomezpoveda and then he fixed it via GitHub's UI, check it out at the issue page :)

[ghost]

@webknjaz can i know if my solution works for you?? and which version are you using?

[webknjaz]

No, I'm just watching this thread for a long time and if I try it at some point, I'll post result.

[ghost]

Okay, I think I've managed to solve every problem!

There are 2 steps to follow.

I have tested the implementations on elephones M2, Galaxy S4 and ZUK.

This seems to solve definitively any problems related to frame size and rotation of the matrix.

Repeat: I ran the bug fix on opencv 3_3_0 (currently the latest version) so I do not know if you can make these changes even to earlier versions of opencv but I suppose it is possible.

STEP 1: JavaCameraView.java

In order to make the frame fullscreen change this line of code FROM:

if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT))
                        mScale = Math.min(((float) height) / mFrameHeight, ((float) width) / mFrameWidth);
                    else
                        mScale = 0;

TO:

if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT))
                        mScale = Math.max(((float) height) / mFrameHeight, ((float) width) / mFrameWidth);
                    else
                        mScale = 0;

STEP 2: CameraBridgeViewBase.java In order to rotate correctly the matrix you can replace deliverAndDrawFrame method with this code:

protected void deliverAndDrawFrame(CvCameraViewFrame frame) {
        Mat modified;

        if (mListener != null) {
            modified = mListener.onCameraFrame(frame);
        } else {
            modified = frame.rgba();
        }

        boolean bmpValid = true;
        if (modified != null) {
            try {
                Utils.matToBitmap(modified, mCacheBitmap);
            } catch(Exception e) {
                Log.e(TAG, "Mat type: " + modified);
                Log.e(TAG, "Bitmap type: " + mCacheBitmap.getWidth() + "*" + mCacheBitmap.getHeight());
                Log.e(TAG, "Utils.matToBitmap() throws an exception: " + e.getMessage());
                bmpValid = false;
            }
        }

        if (bmpValid && mCacheBitmap != null) {
            Canvas canvas = getHolder().lockCanvas();
            if (canvas != null) {
                //this is the rotation part

                if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT && mCameraIndex == -1) {
                    Log.d(TAG, "INDEX IS: " + mCameraIndex);
                    canvas.save();
                    canvas.rotate(90,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT && mCameraIndex == 1) {
                    canvas.save();
                    canvas.rotate(270,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (mScale != 0) {
                    canvas.drawBitmap(mCacheBitmap, new Rect(0,0,mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                            new Rect((int)((canvas.getWidth() - mScale*mCacheBitmap.getWidth()) / 2),
                                    (int)((canvas.getHeight() - mScale*mCacheBitmap.getHeight()) / 2),
                                    (int)((canvas.getWidth() - mScale*mCacheBitmap.getWidth()) / 2 + mScale*mCacheBitmap.getWidth()),
                                    (int)((canvas.getHeight() - mScale*mCacheBitmap.getHeight()) / 2 + mScale*mCacheBitmap.getHeight())), null);
                } else {
                    canvas.drawBitmap(mCacheBitmap, new Rect(0,0,mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                            new Rect((canvas.getWidth() - mCacheBitmap.getWidth()) / 2,
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2,
                                    (canvas.getWidth() - mCacheBitmap.getWidth()) / 2 + mCacheBitmap.getWidth(),
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2 + mCacheBitmap.getHeight()), null);
                }

                if (mFpsMeter != null) {
                    mFpsMeter.measure();
                    mFpsMeter.draw(canvas, 20, 30);
                }
                if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT) {
                    canvas.restore();
                }
                getHolder().unlockCanvasAndPost(canvas);
            }
        }
    }

i'm not sure that mCameraIndex is always -1 for default camera and 1 for front camera.... i'm testing it but i think that's correct! 🗡

Let me know if my solution works for you! 🍡

[ghost]

p.s this solution does not change minimally the framerate :D but the image result more scaled..... trying to resolve it

[jmgomezpoveda]

Roberto: I will try to test this tonight.

On 9 Oct 2017 14:10, "RobertoManfreda" notifications@github.com wrote:

p.s this solution does not change minimally the framerate :D

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencv/opencv/issues/4704#issuecomment-335152942, or mute the thread https://github.com/notifications/unsubscribe-auth/ADjNFOLdgxDcqC-1YtbOi8EaxI9BJKBKks5sqhs3gaJpZM4FgS7v .

[jmgomezpoveda]

On a quick test using a Samsung J5 2017 (with OpenCV 3.2), on portrait I am getting a "java.lang.IllegalStateException: Underflow in restore - more restores than saves" error at canvas.restore() in CameraBridgeViewBase.java. Also, in landscape the preview video appears upside down.

Will do more tests another day, upgrade to 3.3, and report.

[ghost]

ok, probably you have to call canvas.save() after canvas.lock()

[jmgomezpoveda]

Well, in my case mCameraIndex is 0 inside of deliverAndDrawFrame, case which is not currently handled in your proposed function. So replacing the check with:

if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT && mCameraIndex == -1) {
    canvas.save();
    canvas.rotate(90,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
} else if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT && mCameraIndex == 1) {
    canvas.save();
    canvas.rotate(270,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
} else {
    canvas.save();
}

Or the restore with:

if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT &&
        ((mCameraIndex == -1) || (mCameraIndex == 1)) ) {
    canvas.restore();
}

Making either of these changes (but not both!) avoid the crash.

That said, the image is always upside down, in both landscape and portrait.

[ghost]

try logging on mCameraIndex maybe default camera returns 0 and front camera returns 1... Try simplifying the code... remove some statements and build them according to your expectations

[jmgomezpoveda]

Yeah, will do some testing and get back. Ideally we should be able to find a function that works fine everywhere and that we can propose upstream in a PR.

[ghost]

today I've made other code adjustments:

Returning Math.max() was not correct.

I set the mScale in JavaCameraView to 1 like this:

   if ((getLayoutParams().width == LayoutParams.MATCH_PARENT) && (getLayoutParams().height == LayoutParams.MATCH_PARENT))
                        if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT) {
                            mScale = 1;                                                            
                        } else {
                            mScale = 0;
                        }

Then in CameraBridgeViewbase I have transformed this code FROM

Canvas canvas = getHolder().lockCanvas();
            if (canvas != null) {
                canvas.drawColor(0, android.graphics.PorterDuff.Mode.CLEAR);
                if (BuildConfig.DEBUG)
                    Log.d(TAG, "mStretch value: " + mScale);

                if (mScale != 0) {
                    canvas.drawBitmap(mCacheBitmap, new Rect(0, 0, mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                            new Rect((int) ((canvas.getWidth() - mScale * mCacheBitmap.getWidth()) / 2),
                                    (int) ((canvas.getHeight() - mScale * mCacheBitmap.getHeight()) / 2),
                                    (int) ((canvas.getWidth() - mScale * mCacheBitmap.getWidth()) / 2 + mScale * mCacheBitmap.getWidth()),
                                    (int) ((canvas.getHeight() - mScale * mCacheBitmap.getHeight()) / 2 + mScale * mCacheBitmap.getHeight())), null);
                } else {
                    canvas.drawBitmap(mCacheBitmap, new Rect(0, 0, mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                            new Rect((canvas.getWidth() - mCacheBitmap.getWidth()) / 2,
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2,
                                    (canvas.getWidth() - mCacheBitmap.getWidth()) / 2 + mCacheBitmap.getWidth(),
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2 + mCacheBitmap.getHeight()), null);
                }

                if (mFpsMeter != null) {
                    mFpsMeter.measure();
                    mFpsMeter.draw(canvas, 20, 30);
                }
                getHolder().unlockCanvasAndPost(canvas);

TO:

Canvas canvas = getHolder().lockCanvas();
            if (canvas != null) {
                //this is the rotation part
                canvas.save();
                if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT && mCameraIndex == -1) {
                    Log.d(TAG, "INDEX IS: " + mCameraIndex);
                    canvas.rotate(90,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT && mCameraIndex == 1) {
                    canvas.save();
                    canvas.rotate(270,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (mScale != 0) {

                    Rect rect = canvas.getClipBounds();

                    canvas.drawBitmap(mCacheBitmap, new Rect(0,0,mCacheBitmap.getWidth(), mCacheBitmap.getHeight()), rect, null);

                } else {
                    canvas.drawBitmap(mCacheBitmap, new Rect(0,0,mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                            new Rect((canvas.getWidth() - mCacheBitmap.getWidth()) / 2,
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2,
                                    (canvas.getWidth() - mCacheBitmap.getWidth()) / 2 + mCacheBitmap.getWidth(),
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2 + mCacheBitmap.getHeight()), null);

                    Log.d("DIO2", "left"+ (canvas.getWidth() - mCacheBitmap.getWidth()) / 2);
                    Log.d("DIO2", "top"+ (canvas.getHeight() - mCacheBitmap.getHeight()) / 2);
                    Log.d("DIO2", "right"+ (canvas.getWidth() - mCacheBitmap.getWidth()) / 2 + mCacheBitmap.getWidth());
                    Log.d("DIO2", "bottom"+ (canvas.getHeight() - mCacheBitmap.getHeight()) / 2 + mCacheBitmap.getHeight());
                }

                if (mFpsMeter != null) {
                    mFpsMeter.measure();
                    mFpsMeter.draw(canvas, 20, 30);
                }

                if (getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT) {
                    canvas.restore();
                }
                getHolder().unlockCanvasAndPost(canvas);

this code is near to work very great... it need some adjustments on save() and restore() canvas

[ghost]

canvas.getClipBounds do all the work 🗡

[jmgomezpoveda]

I just tested your update and, at least in my Samsung, in one of the landscape orientations the image is fine, but in the other it is upside-down. The same happens with the portrait orientation: one is fine, the other is upside-down.

[ghost]

This seems to be the worke final solution:

 if (bmpValid && mCacheBitmap != null) {
            Canvas canvas = getHolder().lockCanvas();
            if (canvas != null) {
                //this is the rotation part
                //canvas.save();
                Log.d(TAG,"CAMERAINDEX" + mCameraIndex);
                if (getDisplay().getRotation() == Surface.ROTATION_0 && mCameraIndex == -1) {
                    canvas.rotate(90,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if(getDisplay().getRotation() == Surface.ROTATION_0 &&  mCameraIndex == 0){
                    canvas.rotate(90,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (getDisplay().getRotation() == Surface.ROTATION_0 && mCameraIndex == 1) {
                    canvas.rotate(270,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (getDisplay().getRotation() == Surface.ROTATION_270) {
                    canvas.rotate(180,  (canvas.getWidth()/ 2),(canvas.getHeight()/ 2));
                }

                if (mScale != 0) {

                    Rect rect = canvas.getClipBounds();

                    canvas.drawBitmap(mCacheBitmap, new Rect(0,0,mCacheBitmap.getWidth(), mCacheBitmap.getHeight()), rect, null);

                } else {
                    canvas.drawBitmap(mCacheBitmap, new Rect(0,0,mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                            new Rect((canvas.getWidth() - mCacheBitmap.getWidth()) / 2,
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2,
                                    (canvas.getWidth() - mCacheBitmap.getWidth()) / 2 + mCacheBitmap.getWidth(),
                                    (canvas.getHeight() - mCacheBitmap.getHeight()) / 2 + mCacheBitmap.getHeight()), null);
                }

                if (mFpsMeter != null) {
                    mFpsMeter.measure();
                    mFpsMeter.draw(canvas, 20, 30);
                }

                //canvas.restore();
                getHolder().unlockCanvasAndPost(canvas);
            }
        }
    }

We can exclude canvas.save ena restore!

Try :D

[ghost]

I'm running opencv 3_3_0 with adroid build tools 26.0.2.

Hope this will work fine for you too

[crazyscoop]

It works great performance wise. But by rotating the canvas the coordinate system gets rotated.

[jynlix]

really hope this can be merged into the main branch

[morejump]

@RobertoManfreda i tried above solution but camera preview is not full screen, again, i tried it on opencv 3.0.0. I want to full screen both height and with. plz help me!!!

[morejump]

i have small space between preview androi border of my phone.

[d7coders]

@RobertoManfreda the camera rotated but the face detection green square not rotate with the camera

[rajkumarmishra]

solution discussed here: http://answers.opencv.org/question/20325/how-can-i-change-orientation-without-ruin-camera-settings/ https://stackoverflow.com/questions/16669779/opencv-camera-orientation-issue has solved the problem of orientation, working fine for all the devices. but the fps is dropped. can someone give solution to increase fps.

[OctavioCega]

2018 , downloaded OpenCV Android v3.4 (release 23/dec/17) . BUG STILL HERE!

EDIT 07/01/18 @RobertoManfreda so far so good. I have tested your canvas.rotate and it finally shows portrait. Since I only need portrait mode everything is ok but I think that there is a problem with scale. EDIT 2 07/01/18 Something is wrong with coordinate system. Now, I cant see FPS meter. I dont think that canvas.rotate is final solution. We need help from alalek EDIT 3 11/01/2018

I had an idea. After finding the part when opencv takes picture from camera my goal was to rotates the camera before opencv process. I've modified without success the javaviewcamera.java (opencv 3.4) with this code:

LINE 193
mBuffer = new byte[size];
mCamera.setDisplayOrientation(90); //insert this
mCamera.addCallbackBuffer(mBuffer);

LINE 207
//                    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB) {
//                        mSurfaceTexture = new SurfaceTexture(MAGIC_TEXTURE_ID);
//                        mCamera.setPreviewTexture(mSurfaceTexture);
//                    } else
//
//                       mCamera.setPreviewDisplay(null);
    mCamera.setPreviewDisplay(getHolder()); //insert this

It successfully rotates to full screen portrait mode but EdgeCannyDetectors does nothing at onCameraFrame() I think that mCamera.setPreviewDisplay(getHolder()); changes the way of delivering frame. I'm still using @RobertoManfreda solution :(

Edit 4 11/01/2018 2:19am I think I achieve a solution....testing later. If it works I will post it here. 3:00am It works!!

[OctavioCega]

Here is a solution, I've tested at Samsung Galaxy Edge S7, Moto X 2016, Lenovo Tablet and Alcatel One Touch. The camera preview is Full Screen, well scalated PORTRAIT MODE. I was finding a way to intercept the camera info before OpenCV gets it. So I went to CameraBridgeViewBase.java (OpenCv 3.4)

Matrix matrix = new Matrix(); // I rotate it with minimal process
matrix.preTranslate((canvas.getWidth() - mCacheBitmap.getWidth()) / 2,(canvas.getHeight() - mCacheBitmap.getHeight()) / 2);
matrix.postRotate(90f,(canvas.getWidth()) / 2,(canvas.getHeight()) / 2);
float scale = (float) canvas.getWidth() / (float) mCacheBitmap.getHeight();
matrix.postScale(scale, scale, canvas.getWidth()/2 , canvas.getHeight()/2 );
canvas.drawBitmap(mCacheBitmap, matrix, new Paint());

We need to insert that after LINE 415 (Code for reference) :

if (bmpValid && mCacheBitmap != null) {
    Canvas canvas = getHolder().lockCanvas();
    if (canvas != null) {
        canvas.drawColor(0, android.graphics.PorterDuff.Mode.CLEAR);
        //insert here
    }
}

What do you think? It works very well and only loose 1-2fps. Any feedback? I've spent several days because I need realtime portrait mode preview :D NOTE: the only bug I've found so far is that Imgproc.putText is 90 degree turned.

[ghost]

Ok, hi all... Sorry if i wasn't here but i had some small problems... Now i'm ready to solve this issue with you! @OctavioCega i noticed you that i tried your solution but it is not full screen, and my old solution had some scale problems. Now i'm here with a new solution but i need your help because this solution works, with no scale problem etc but fps dropping dramatically.

I'm using latest version of android pack opencv (3.4.0) downloadable at this link https://opencv.org/releases.html

Now we are ready to start.

First of all, i'm testing this solution working with front camera because at the moment my app only needs the front camera, so i go in my xml (where we delcare JavaCameraView) and i use this code

<org.opencv.android.JavaCameraView
        android:id="@+id/javaCameraViewXml"
        android:layout_width="fill_parent"
        android:layout_height="fill_parent"
        android:visibility="gone"
        opencv:camera_id="front"
        opencv:show_fps="true" />

using opencv:camera_id="front" will start the application using front camera.

Go to CameraBridgeViewBase. java and under canvas.drawColor(0, PorterDuff.Mode.CLEAR); , insert this code

if (canvas != null) {

                canvas.drawColor(0, PorterDuff.Mode.CLEAR);

                canvas.rotate(270, (canvas.getWidth() / 2), (canvas.getHeight() / 2));
                canvas.scale(1, -1, canvas.getWidth() / 2, canvas.getHeight() / 2);

                Log.d(TAG, "Accelerationn " + canvas.isHardwareAccelerated());

                canvas.drawBitmap(mCacheBitmap, new Rect(0,0, mCacheBitmap.getWidth(), mCacheBitmap.getHeight()),
                        new Rect((canvas.getWidth() - mCacheBitmap.getWidth()) / 2,
                                (canvas.getHeight() - mCacheBitmap.getHeight()) / 2,
                                (canvas.getWidth() - mCacheBitmap.getWidth()) / 2 + mCacheBitmap.getWidth(),
                                (canvas.getHeight() - mCacheBitmap.getHeight()) / 2 + mCacheBitmap.getHeight()),
                        null);   

                if (mFpsMeter != null) {
                    mFpsMeter.measure();
                    mFpsMeter.draw(canvas, 20, 30);
                }

                getHolder().unlockCanvasAndPost(canvas);
}

to solve orientation and reflection problems! I removed mScale field and his relative calls becdause i'm trying to work without scale! Now the image will result ok.

At this point we have another problem: the image is not full-screen!

So go in JavaCameraView.java and find the try-catch block (line 142). I modified the code as follows:

try {
                Camera.Parameters params = mCamera.getParameters();
                Log.d(TAG, "getSupportedPreviewSizes()");
                List<android.hardware.Camera.Size> sizes = params.getSupportedPreviewSizes();

                if (sizes != null) {
                    /* Select the size that fits surface considering maximum size allowed */
                    Size frameSize = calculateCameraFrameSize(sizes, new JavaCameraSizeAccessor(), width, height);

                    /* Image format NV21 causes issues in the Android emulators */
                    if (Build.FINGERPRINT.startsWith("generic")
                            || Build.FINGERPRINT.startsWith("unknown")
                            || Build.MODEL.contains("google_sdk")
                            || Build.MODEL.contains("Emulator")
                            || Build.MODEL.contains("Android SDK built for x86")
                            || Build.MANUFACTURER.contains("Genymotion")
                            || (Build.BRAND.startsWith("generic") && Build.DEVICE.startsWith("generic"))
                            || "google_sdk".equals(Build.PRODUCT))
                        params.setPreviewFormat(ImageFormat.YV12);  // "generic" or "android" = android emulator
                    else
                        params.setPreviewFormat(ImageFormat.NV21);

                    mPreviewFormat = params.getPreviewFormat();

                    params.setPreviewSize(1920, 1080);

                    mCamera.setParameters(params);
                    params = mCamera.getParameters();

                    mFrameWidth = 1920;
                    mFrameHeight = 1080;

                    if (mFpsMeter != null) {
                        mFpsMeter.setResolution(mFrameWidth, mFrameHeight);
                    }

                    int size = mFrameWidth * mFrameHeight;
                    size  = size * ImageFormat.getBitsPerPixel(params.getPreviewFormat()) / 8;
                    mBuffer = new byte[size];

                    mCamera.addCallbackBuffer(mBuffer);
                    mCamera.setPreviewCallbackWithBuffer(this);

                    mFrameChain = new Mat[2];
                    mFrameChain[0] = new Mat(mFrameHeight + (mFrameHeight/2), mFrameWidth, CvType.CV_8UC1);
                    mFrameChain[1] = new Mat(mFrameHeight + (mFrameHeight/2), mFrameWidth, CvType.CV_8UC1);

                    AllocateCache();

                    mCameraFrame = new JavaCameraFrame[2];
                    mCameraFrame[0] = new JavaCameraFrame(mFrameChain[0], mFrameWidth, mFrameHeight);
                    mCameraFrame[1] = new JavaCameraFrame(mFrameChain[1], mFrameWidth, mFrameHeight);

                    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB) {
                        mSurfaceTexture = new SurfaceTexture(MAGIC_TEXTURE_ID);
                        mCamera.setPreviewTexture(mSurfaceTexture);
                    } else
                       mCamera.setPreviewDisplay(null);

                    /* Finally we are ready to start the preview */
                    Log.d(TAG, "startPreview");
                    mCamera.startPreview();
                }
                else
                    result = false;
            } catch (Exception e) {
                result = false;
                e.printStackTrace();
            }

PLEASE NOTE: i used 1920 and 1080 values because this is my screen resolution, obviously we have to find resolution using getSupportedPreviewSizes(), so for the moment change that values with your screen resolution! Then we'll find a correct method to automatically select the best resolution.

Another info for you: Using this "hack" will cause fps dropping, at the moment i'm trying to enable hardware acceleration so i can see if there will be some improvement.

Consider this like a raw solution, we have to do some improvements.

At the same time i'll try what happens if i set a minimum resolution in JavaCameraView.java for example starting from 640x480 or less then scaling bitmap in CameraBridgeViewBase.java after locking the canvas object.

Now try this solution and tell me what is the result. I tested it only on my smartphone so i can't know if this solution works for all!

Test and post here, we use open-source code so we have to give some improvement and some help to original developers. Thank you all, waiting for you 🗡

EDIT I forgot to tell you that with my changes fpsmeter disappears from the display because we have mirrored and rotated the image, so search for "fps" in your logcat

[ghost]

We are very near to solve this bug... I'm working on it...

Now i tell you latest changes:

CameraBridgeViewBase.java deliverAndDrawFrame(CVCameraViewFrame frame):

 protected void deliverAndDrawFrame(CvCameraViewFrame frame) {
        Mat modified;

        if (mListener != null) {
            modified = mListener.onCameraFrame(frame);
        } else {
            modified = frame.rgba();
        }

        boolean bmpValid = true;
        if (modified != null) {
            try {
                Utils.matToBitmap(modified, mCacheBitmap);
            } catch(Exception e) {
                Log.e(TAG, "Mat type: " + modified);
                Log.e(TAG, "Bitmap type: " + mCacheBitmap.getWidth() + "*" + mCacheBitmap.getHeight());
                Log.e(TAG, "Utils.matToBitmap() throws an exception: " + e.getMessage());
                bmpValid = false;
            }
        }

        mFpsMeter.measure();
    }

JavaCameraView.java initializeCamera(int width, int height)

protected boolean initializeCamera(int width, int height) {
        Log.d(TAG, "Initialize java camera");
        boolean result = true;
        synchronized (this) {
            mCamera = null;

            if (mCameraIndex == CAMERA_ID_ANY) {
                Log.d(TAG, "Trying to open camera with old open()");
                try {
                    mCamera = Camera.open();
                }
                catch (Exception e){
                    Log.e(TAG, "Camera is not available (in use or does not exist): " + e.getLocalizedMessage());
                }

                if(mCamera == null && Build.VERSION.SDK_INT >= Build.VERSION_CODES.GINGERBREAD) {
                    boolean connected = false;
                    for (int camIdx = 0; camIdx < Camera.getNumberOfCameras(); ++camIdx) {
                        Log.d(TAG, "Trying to open camera with new open(" + Integer.valueOf(camIdx) + ")");
                        try {
                            mCamera = Camera.open(camIdx);
                            connected = true;
                        } catch (RuntimeException e) {
                            Log.e(TAG, "Camera #" + camIdx + "failed to open: " + e.getLocalizedMessage());
                        }
                        if (connected) break;
                    }
                }
            } else {
                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.GINGERBREAD) {
                    int localCameraIndex = mCameraIndex;
                    if (mCameraIndex == CAMERA_ID_BACK) {
                        Log.i(TAG, "Trying to open back camera");
                        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
                        for (int camIdx = 0; camIdx < Camera.getNumberOfCameras(); ++camIdx) {
                            Camera.getCameraInfo( camIdx, cameraInfo );
                            if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_BACK) {
                                localCameraIndex = camIdx;
                                break;
                            }
                        }
                    } else if (mCameraIndex == CAMERA_ID_FRONT) {
                        Log.i(TAG, "Trying to open front camera");
                        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
                        for (int camIdx = 0; camIdx < Camera.getNumberOfCameras(); ++camIdx) {
                            Camera.getCameraInfo( camIdx, cameraInfo );
                            if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
                                localCameraIndex = camIdx;
                                break;
                            }
                        }
                    }
                    if (localCameraIndex == CAMERA_ID_BACK) {
                        Log.e(TAG, "Back camera not found!");
                    } else if (localCameraIndex == CAMERA_ID_FRONT) {
                        Log.e(TAG, "Front camera not found!");
                    } else {
                        Log.d(TAG, "Trying to open camera with new open(" + Integer.valueOf(localCameraIndex) + ")");
                        try {
                            mCamera = Camera.open(localCameraIndex);
                        } catch (RuntimeException e) {
                            Log.e(TAG, "Camera #" + localCameraIndex + "failed to open: " + e.getLocalizedMessage());
                        }
                    }
                }
            }

            if (mCamera == null)
                return false;

            /* Now set camera parameters */
            try {
                Camera.Parameters params = mCamera.getParameters();
                Log.d(TAG, "getSupportedPreviewSizes()");
                List<android.hardware.Camera.Size> sizes = params.getSupportedPreviewSizes();

                if (sizes != null) {
                    /* Image format NV21 causes issues in the Android emulators */
                    if (Build.FINGERPRINT.startsWith("generic")
                            || Build.FINGERPRINT.startsWith("unknown")
                            || Build.MODEL.contains("google_sdk")
                            || Build.MODEL.contains("Emulator")
                            || Build.MODEL.contains("Android SDK built for x86")
                            || Build.MANUFACTURER.contains("Genymotion")
                            || (Build.BRAND.startsWith("generic") && Build.DEVICE.startsWith("generic"))
                            || "google_sdk".equals(Build.PRODUCT))
                        params.setPreviewFormat(ImageFormat.YV12);  // "generic" or "android" = android emulator
                    else
                        params.setPreviewFormat(ImageFormat.NV21);

                    mPreviewFormat = params.getPreviewFormat();
                    if (!Build.MODEL.equals("GT-I9100")) params.setRecordingHint(true);
                    params.setPreviewSize(1920, 1080);
                    mCamera.setParameters(params);

                    mFrameWidth = 1920;
                    mFrameHeight = 1080;

                    if (mFpsMeter != null) {
                        mFpsMeter.setResolution(mFrameWidth, mFrameHeight);
                    }

                    int size = mFrameWidth * mFrameHeight;
                    size  = size * ImageFormat.getBitsPerPixel(params.getPreviewFormat()) / 8;
                    mBuffer = new byte[size];

                    mCamera.addCallbackBuffer(mBuffer);
                    mCamera.setPreviewCallbackWithBuffer(this);

                    mFrameChain = new Mat[2];
                    mFrameChain[0] = new Mat(mFrameHeight + (mFrameHeight/2), mFrameWidth, CvType.CV_8UC1);
                    mFrameChain[1] = new Mat(mFrameHeight + (mFrameHeight/2), mFrameWidth, CvType.CV_8UC1);

                    AllocateCache();

                    mCameraFrame = new JavaCameraFrame[2];
                    mCameraFrame[0] = new JavaCameraFrame(mFrameChain[0], mFrameWidth, mFrameHeight);
                    mCameraFrame[1] = new JavaCameraFrame(mFrameChain[1], mFrameWidth, mFrameHeight);

                    mSurfaceTexture = new SurfaceTexture(MAGIC_TEXTURE_ID);
                    mCamera.setPreviewTexture(mSurfaceTexture);

                    if (getOrientation().equals("portrait")) {
                        setDisplayOrientation(mCamera, 90);
                    } else if (getOrientation().equals("reverse landscape")){
                        setDisplayOrientation(mCamera, 180);
                    }
                    mCamera.setPreviewDisplay(getHolder());

                    mCamera.startPreview();
                }
                else
                    result = false;
            } catch (Exception e) {
                result = false;
                e.printStackTrace();
            }
        }

        return result;
    }

Added two methods: setDisplayOrientation(Camera camera, int angle) and getOrientation(). Thanks to user2235615 for posting the first one it on stackOverflow https://stackoverflow.com/questions/16669779/opencv-camera-orientation-issue.

    private void setDisplayOrientation(Camera camera, int angle){
        Method downPolymorphic;
        try {
            downPolymorphic = camera.getClass().getMethod("setDisplayOrientation", int.class);
            if (downPolymorphic != null) {
                downPolymorphic.invoke(camera, angle);
            }
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

    private String getOrientation(){
        int orientation = Surface.ROTATION_0;

        WindowManager wm = (WindowManager) getContext().getSystemService(Context.WINDOW_SERVICE);
        if (wm != null) {
            Display display = wm.getDefaultDisplay();
            orientation = display.getOrientation();
        }

        if (orientation == Surface.ROTATION_0) return "portrait";
        if (orientation == Surface.ROTATION_90) return "landscape";
        else return "reverse landscape";
    }

Now i'm finding a method that returns ALWAYS the biggest resolution for preview.

This solution finally works for back and front camera and fps are 17-18 on my elephone m2.

[gigadeplex]

@RobertoManfreda your solution looks nice however, now i can't use onCameraFrame.. for example if i want to change my camera view to grayscale, it won't change. any ideas on how to fix this?

[ZenLiuCN]

mCamera.setPreviewDisplay(getHolder()); //this is the key point, not use mSurfaceTexture to display ,so can't be modifiy on Camera preview so RobertoManfreda 's patch just useless, because it just directly use Camera to control drawing,opencv can do nothing on each frame

[Tarapaty]

Hello I'm trying to make this work for me and so far the best solution i found is presented in https://github.com/duddns/Andorid-OpenCV-Portrait-Camera But this solution reduced frame rate a lot and you have point 0,0 in top right corner.

I also added resolution choosing based on devices aspect ratio to get rid of black edges in preview. But more then this I don't have a clue what i can do to make it work smoother

[sencagri]

I just started a opencv project on Android and shocked when i faced up to this problem. It is crucial bug that has to be solved immediately. Is there a solution that does not effect fps?

[ghost]

There is a solution, i proposed it. The Only solution that doesn't affect FPS is using that reflective method. But with this approach we'll be unable to manage with the output mat! So i can use this solution only when i don't want to modify the mat obj. Very unprobable!

[smelzer]

@RobertoManfreda, your solution worked like a champ on my Samsung SM-G900V and OpenCV 3.4.2.. I will test on a few more devices, but I want to thank you for your hard work on this. In my case I do not want to modify the preview but rather capture and process the frames.

Is there a reason the above code is not just added to the master branch code base? If there are other issues that I have not come across yet, I would think it would be better to resolve as part of the master branch and have a single universal solution to orientation.

[ghost]

I don't know why they are not pushing my solution to the master branch! I think thtat they are searching for a finally and 100% working solution... I'm not working on opencv at the moment, but i'll use It in a future, sure! So i Hope that we'll find a solution. Opencv is very useful and powerful but this bug is so subtle! Always happy to help! Happy coding guys :dagger:

[shingohu]

@RobertoManfreda why the camera Preview is too blurry for me i think you forget the camera focus code

                List<String> FocusModes = params.getSupportedFocusModes();
                if (FocusModes != null && FocusModes.contains(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO)) {
                    params.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);
                }

[romanshemsheichariot]

Any progress?

[sencagri]

Nothing changed

Roman Shemshei notifications@github.com, 19 Mar 2019 Sal, 20:48 tarihinde şunu yazdı:

Any progress?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opencv/opencv/issues/4704#issuecomment-474492282, or mute the thread https://github.com/notifications/unsubscribe-auth/AHlSRTsId2lU6tAcbnVnEoe6GxylVnoPks5vYSMAgaJpZM4FgS7v .

-- Ebubekir Çağrı ŞEN

[alexndesousa]

@OctavioCega your method worked perfectly for me. Thanks a lot.

[topherbuckley]

I get no significant decrease in framerate by simply calling // Insert appropriate flipCode here Core.flip(super.mRgba, super.mRgba,1); within my overridden onCameraFrame() method

This is documented here See Core class then search for flip or flipCode.

[jslok]

@hugolm84 I tried you latest code but it seems the line mCamera.setPreviewDisplay(getHolder()); prevents any image processing from onCameraFrame() from showing on the preview just like in the first suggestion by @OctavioCega . Is that not the case for you?