Camera view is sideways on Android

[jslok]

On Android, CvCamera view is rotated 90 degrees counterclockwise. I tried several solutions I found during search including setting android orientation to portrait in androidmanifest.xml and adding Core.rotate(in, in, 0) in CvCameraView.java on line 454. Core.rotate just causes the app to crash. Core.flip works but only flips the view 180 degrees.

[adamgf]

I will fix this issue can you please give me some more info what type of Android device etc.?

[jslok]

I use a galaxy s9 with app in portrait mode. I think this would happen on all devices though. I checked your tests (thanks for excellent examples) and they set orientation to landscape in androidmanifest so the cvcamera does appear correct. Very much appreciste your work with this module!

[adamgf]

Thank you @jslok! Oh okay I never added the code to change the orientation I wanted to get this module to a point where the react-native & opencv people could get involved and finish it off I worked on it for months! I am going to be working on it for at least another three weeks and then putting together the 1.0.5 release with documentation et. al. I will look into the rotation issue for you and see if I can add a parameter to CvCamera. You can always use <CvCamera ... /> and call a rotation function from Core or ImgProc as long as the function is part of Core or ImgProc it should be invoked at runtime on Android but may not be supported yet on iOS. That is the main thing I need to do to finish this is implement the Core and ImgProc wrapper functions on iOS.

[adamgf]

I don't know why that did not show but it is around <CvCamera ... /> and then a I do not remember the exact function name or parameters the second and third parameters should be the Mat to rotate I think

[adamgf]

The CvInvoke tag is not showing up for some reason when I post these messages

[jslok]

Thanks for the pointer @adamgf.

I tried with some trial and error with cvinvoke based off code from your tests and it seems flip and rotate functions 180 degrees works but I cannot get rotate 90 degrees to show. It just doesn't show the inner window. I am not experienced with opencv to begin with so maybe I am doing something wrong. Just starting to learn.

      <View style={styles.container}>
        <CvInvokeGroup groupid="invokeGroup0">
          <CvInvoke inobj="rgbaInnerWindow" func="release" />
          <CvInvoke
            func="rotate"
            params={{
              p1: "rgbaInnerWindow",
              p2: "rgbaInnerWindow",
              p3: 1 //0 and 2 don't work
            }}
          />
          <CvInvoke
            inobj="rgba"
            func="submat"
            params={{ p1: top, p2: bottom, p3: left, p4: right }}
            outobj="rgbaInnerWindow"
          />
          {this.renderCamera()}
        </CvInvokeGroup>
      </View>

[jslok]

I'll await the documentation with the 1.05 release. I have many questions but I'm sure the documentation will handle most. Thanks!

[adamgf]

Hello @jslok I am going to start checking these comments daily my bad I apologize! I just had my birthday so I took a couple of days off around that but Iam back on it. First of all I do not think you have to create a submat and then release it you can just rotate the mat in place.

Zero should work if it does not then there must be a bug in the code somewhere.

enum RotateFlags { ROTATE_90_CLOCKWISE = 0, //Rotate 90 degrees clockwise ROTATE_180 = 1, //Rotate 180 degrees clockwise ROTATE_90_COUNTERCLOCKWISE = 2, //Rotate 270 degrees clockwise };

I willl look into it some more this is definitely not a super stable solution yet there could be some bugs I am the only one who has worked on this.

[adamgf]

Just wanted to let everybody know that the v1.0.5 version has been released I did fix the sample app SimpleGreyscale to rotate counterclockwise with this release so you may want to check it out. Spread the word!

[jslok]

@adamgf I'm having some trouble with the orientation still unfortunately. Your readme say to put the app in landscape mode in the AndroidManifest.xml but this is not possible for my app. It's a portrait mode app.

I checked the SimpleGreyscale to try and adapt it for the CvCamera but it doesn't seem to work. Do I have something wrong here? <CvInvokeGroup groupid="zeeGrup"> <CvInvoke func="Rotate" params={{p1: 'dstMat', p2: 'dstMat', p3: Core.ROTATE_90_CLOCKWISE}} /> <CvCamera ref={ref => { this.cvCamera = ref; }} style={styles.preview} facing={'back'} /> </CvInvokeGroup>

If I use the CameraPreview example and change only landscape to portrait in AndroidManifest.xml, the preview is small, centered in the middle of the screen, and sideways. I've also tried setting cameraView.setMaxFrameSize(1080, 1920); in CvCameraModule.java ln 64 but it does not enlarge the frame. I can only shrink the frame using that line.

This is how it looks:

Any help you can provide or pointers would be much appreciated. Thanks!

[adamgf]

I will try modifying the CvCameraPreview app to get this to work for you and remove the landscape property from AndroidManifest.xml and then update this thread with the solution. There should be a way to do it using the existing package.

[adamgf]

Oh I just noticed something 'p1' value should be set to 'rgba' so <CvInvoke func="rotate" params={{p1: 'rgba', p2: 'dstMat', p3: Core.ROTATE_90_CLOCKWISE}} />

[adamgf]

That did not work let me look into this some more and update the thread once I figure how to do this correctly I may have to update the package to a 1.0.8 version if back-end changes are needed.

[jslok]

Hi @adamgf I've been trying to fix this for the past couple days and I think I learned a lot. It seems to be an issue with opencv android itself. I think rotating or flipping 90 degrees does not work at all for the camera preview (crashes immediately) because onCameraFrame must return a Mat of the same exact dimensions.

I tried every single suggestion I could google and this thread so far has been the most help by using a combination of code changes. https://github.com/adamgf/react-native-opencv3/issues/4

So far this is the best I've been able to get with these two code changes:

1. Override deliverAndDrawFrame from CameraBridgeViewBase.java and add canvas.rotate(90, (canvas.getWidth() / 2), (canvas.getHeight() / 2)); after canvas.drawColor(0, android.graphics.PorterDuff.Mode.CLEAR);. This rotates the image to be portrait, but it is not full screen.

2. Override calculateCameraFrameSize and swap values of maxAllowedWidth and maxAllowedHeight. This correction enlarges the frame to as a big as possible. Opencv is still using width/height as if it were landscape so width must be the larger value.

With those changes it shows portrait full screen but FPS is decreased a lot, from 30 fps to 15-20, since it must rotate every frame. Imgproc.puttext() are all still sideways but not too big of a deal. I think opencv stays with width being the y values and height is x. Obviously we can elaborate on these changes to first check the orientation before making the changes.

Ideally just adding mCamera.setDisplayOrientation(90); mCamera.setPreviewDisplay(getHolder()); to initializeCamera would be best and does not affect the fps much if any, but the line mCamera.setPreviewDisplay(getHolder()); seems to prevent any image processing in onCameraFrame from showing on the preview at all.

I wonder if it is fixed in the 4.1.0-contrib version. I'm not sure about the changes and I tried just swapping out to the higher version in build.gradle but the app wouldn't start at all. I don't know enough about the java side to troubleshoot it. I'm really surprised to have come across this issue. Are most people writing apps in landscape mode? I don't know about full native apps but I think most react-native apps are probably portrait.

I also tried using the camera2 api instead by having CvCameraView extend JavaCamera2View instead of JavaCameraView but ran into some issues I don't know how to troubleshoot again. No idea if camera2 api fixes this for sure though.

Really appreciate your help with this!

[adamgf]

Hi @jslok thank you for the info I will spend some time next week trying to get to the bottom of this. I think I made you a contributor if you do want to incorporate something feel free to make any changes. Once I figure something out and have something to contribute I will ping you right now I am not really sure how to fix this yet it sounds like there could be issues with the OpenCV Android library or something else weird going on but I do know enough about how this works I should be able to get some more info together after some further investigation.

[adamgf]

You shouldn't have to force your app to be in landscape mode just to use this package.

[jslok]

Just noticed I didn't paste the right link earlier. This is the thread that gave the most help for me. https://github.com/opencv/opencv/issues/4704

[adamgf]

Thanks @jslok I still have not gotten to this like I wanted to last week but I am definitely spending some cycles on it this weekend and will post an update when I have something to add. Thanks for the thread reference I will definitely use that. -- Adam

[adamgf]

Hi @jslok I figured out what the issue is and the solution basically it is going to require another release of react-native-opencv3. It is going to take a little while to do another release becaues all the sample apps need to be re-unit tested. I will do the release pretty soon and let you know once it is available. The issue is basically that there was a bug in react-native-opencv3 such that if the src and dst are both set to 'rgba' it was not updating the dst Mat correctly. 'srcMat' and 'dstMat' should only be used for 'CvImage.' This should be more robust but ...

[adamgf]

So once I push the next release it should be as:

<CvInvokeGroup groupid='zeeGrup'>
  <CvInvoke func='rotate' params={{"p1":"rgba","p2":"rgba","p3":Core.ROTATE_90_CLOCKWISE}}/>        
  <CvCamera
    ref={ref => { this.cvCamera = ref }}
    style={styles.preview}
    facing={facing}
    useStorage={true}
  />
</CvInvokeGroup>

[jslok]

Awesome, looking forward to trying it out!

[adamgf]

Hi @jslok I tried to fix this and I spent a few days on it and I just couldn't figure it out. It is really tricky and I just don't know Android and React Native well enough. I put a lot of work into this package to contirubte something cool to open source but at this point I kind of don't want to keep working on it. I believe I added you as a contributor if you want to take another crack at it.

[Tex2wo]

I think the community run 'React Native Camera' is also having issues with orientation

https://github.com/react-native-community/react-native-camera/issues/2638

[adamgf]

I want this to be community run. I don't want to run it any more. I wrote it for the sole purpose of it being used by other people not myself.

[nolife08021]

@jslok @adamgf i facing same sideways issue, may I know is there any solution or workaround for this?