Android preview and picture taken show different scale/zoom/dimensions

[Murilo-Perrone]

I can't take pictures properly in my Nativescript-Android project using nativescript-camera-plus. I have tried everything possible from my side. Might be a bug in fancycamera component?

Up to v3.0.8 (uses fancycamera 1.2.3):

• The camera preview (before the picture is taken) is zoomed (aprox 2x zoom), hiding a large extension of viewable area.
• The picture taken then showed a much larger area (which is correct, the maximum area the camera can capture).

Now in v3.1.0 (uses fancycamera 3.0.0-alpha13):

• The camera preview now is working OK, showing the full visible area.
• However, the picture taken now gets cropped in top and bottom, until the ratio is inverted (3:4 to 4:3).

For instance, I'm using portrait 3:4 view, expecting a portrait picture with maximum resolution (3024x4032 photo). But v3.1.0 crops it down into a 3024x2268 landscape image.

Sample code:

import { screen, ScreenMetrics } from "tns-core-modules/platform/platform";
...
  ngOnInit(): void {
    const m: ScreenMetrics = screen.mainScreen;
    this.metrics = {
      widthDIPs: m.widthDIPs,
      heightDIPs: m.heightDIPs,
      ...
    };
  }

        <CameraPlus
            width="{{metrics.widthDIPs}}"
            height="{{metrics.widthDIPs * (4/3)}}"
            ...
            (photoCapturedEvent)="photoCapturedEvent($event)"
            (loaded)="camLoaded($event)">
        </CameraPlus>

Logs:

Camera width: 392.72727272727275, height:523.6363636363636, scaleX: 1, scaleY:1
Screen metrics:
{
  "widthPixels": 1080,
  "heightPixels": 2160,
  "widthDIPs": 392.72727272727275,
  "heightDIPs": 785.4545454545455,
  "scale": 2.75
}

Is there any workaround? I noticed the same problem also affects demo-ng...

[triniwiz]

By default the camera uses the preview ratio 4:3 now , it will choose the largest photo size of 4:3 available for the current camera on the device however you can now customize both.

First you will need to get the supported preview ratios by calling cameraPlusInstance.getGetSupportedRatios() it returns an array of the supported preview ratios which can be passed to cameraPlusInstance.ratio next you will need to set a pictureSize of your liking or allow the system will use the largest size of the chosen ratio .

To get an array of the picture sizes the currently open camera supports you will need to call the following passing the ratio you previously got cameraPlusInstance.getAvailablePictureSize(currentRatio) it will return an array of the supported picture sizes of that ratio that u can set cameraPlusInstance.pictureSize <- also you can get the current pictureSize

[Murilo-Perrone]

Thanks for your quick response. I had already explored those new methods, though, and they didn't help. Because the plugin won't accept a ratio like 3:4 where width < height.

    const ratio = "3:4";
    this.cam.ratio = ratio; // won't store this value
    this.cam.pictureSize = "3024x4032"; // value not accepted again
    const size = this.cam.getAvailablePictureSizes(ratio)[0];
    this.cam.pictureSize = size.height + "x" + size.width; // trying to invert them, no luck

Yet I'm actually just using a regular 4:3 rotated because of phone's position in PORTRAIT mode. The preview seems to understand that, but the image capture doesn't. It's just a regular landscape image in default ratio rotated -90 degrees, just like the phone's original camera does:

I even tried to set the orientation directly in fancycam:

    const fancycamera = (<any>this.cam)._camera;
    fancycamera.setCameraOrientation(com.github.triniwiz.fancycamera.CameraOrientation.PORTRAIT); // value is accepted, but that won't fix the problem
    fancycamera.setRatio("3:4"); // won't accept that value

Where is the code that does this image crop? Is it in fancycam or in android SDK?

[triniwiz]

it's behaving as it should you are doing the correct thing but I missed updating the internal call from that method here that needs to call cameraView.rotation = orientation you should be able to access the internal class by doing this._camera.getChildAt(0) then calling .setRotation(com.github.triniwiz.fancycamera.CameraOrientation.PORTRAIT) it will take the photo as you like also you should use the fancycamera.setRatio("4:3");

    const fancycamera = (<any>this.cam)._camera;
    const internalCamera = fancycamera.getChildAt(0);
    internalCamera.setCameraOrientation(com.github.triniwiz.fancycamera.CameraOrientation.PORTRAIT); // value is accepted, and should take effect
    fancycamera.setRatio("4:3"); 

I'll try getting a new release out with the fix also the cropping is done here and here

[Murilo-Perrone]

Thanks again for your quick response. I found a way to workaround it and a way to fix fancycamera code:

1) WORKROUND:

The internal camera does not have a cameraOrientation method. Instead, has the rotation: CameraOrientation property, and the currentOrientation: Int property which I can't change from nativescript because it is market as internal. Setting only the rotation property in the internal camera did avoid the cropping issue:

const fancycamera = (<any>this.cam)._camera;
const internalCamera = fancycamera.getChildAt(0);
internalCamera.setRotation(com.github.triniwiz.fancycamera.CameraOrientation.PORTRAIT);

The drawback is that the image I got has a 90 degree rotation and position adjustments, ending up with this complex visual workround:

<StackLayout left="-65.5" top="65.5" width="{{previewHeight}}" height="{{previewWidth}}"
  class="cover-image" style="background-image: url('{{capturedImgPath}}')">

.cover-image {
    position: absolute;
    object-fit: cover;
    rotate: 90;
    background-position: center;
    background-size: 100% 100%;
}

2) FIX:

So I managed to fix fancycamera code to make it work properly. To test it, I copied classes.jar into nativescript-camera-plus:

cp ./fancycamera/build/intermediates/compile_library_classes_jar/release/classes.jar  ../your-camera-project/

And changed nativescript-camera-plus gradle file in node_modules/@nstudio/nativescript-camera-plus/platforms/android/include.gradle of my project:

allprojects {
    repositories {
        maven { url "https://jitpack.io" }
        google()
        maven { url "https://dl.bintray.com/triniwiz/maven" }
        flatDir {
            dirs '/.../.../my-project-name' // ADD YOUR PROJECT's FULL PATH HERE
        }
    }
}
dependencies {
    implementation 'androidx.multidex:multidex:2.0.1'
    compile(name:'classes', ext:'jar') {
        transitive = true
    }
    api "androidx.core:core:1.3.2"
    api "androidx.camera:camera-core:1.0.0-beta12"
    api "androidx.camera:camera-camera2:1.0.0-beta12"
    api "androidx.camera:camera-view:1.0.0-alpha19"
    api "androidx.camera:camera-lifecycle:1.0.0-beta12"
}

also implementation "org.jetbrains.kotlin:kotlin-stdlib:1.4.20" for demo projects. See all possible dependencies here.

3) Discussion:

So it seems the cropping was not done by fancycamera but rather by android's ImageCapture class. Fancycamera has code for in-memory rotation steps ( here ), though it is only used when autoSquareCrop: true, which isn't the case here (I don't want a squared image). So I opened this issue.

[Murilo-Perrone]

Fixes are on the way (fancycam#12). They work with app that starts in any of the 4 possible orientations (portrait, landscape, landscape reversed or portrait reversed).

However, for a unlocked app, the switching between orientations in runtime is not yet. That's because the App "flips" aren't being detected and thus fancycamera isn't reacting to them. Fancycamera uses a OrientationEventListener, but that is not what we need, since it's independent from the view rotation.

A client workaround is to map "orientationChanged" event, which can be tried in the demo code:

constructor(...) {
  on("orientationChanged",  (evt) => {
    setTimeout(() => this.onOrientationChanged(evt.eventName, evt.newValue), 1000) // Restart preview
  });
}
onOrientationChanged(eventName: string, newValue: string) {
  if (isAndroid && this.cam) {
    const fancyCamera = (<any>this.cam)._camera as com.github.triniwiz.fancycamera.FancyCamera;
    fancyCamera.stopPreview();
    setTimeout(() => fancyCamera.startPreview(), 1000);
  }

Tried to move this into the plugin's code n initNativeView method, but the orientation event won't be triggered when mapped from there. Then I noticed there is an alternative event registered in camera-plus.android.ts:

initNativeView() {
  super.initNativeView();
  this.on(View.layoutChangedEvent, this._onLayoutChangeListener);

Oddly enough, the "layoutChanged" event was also not being triggered for Android/angular. And mapping this event from client code also didn't work. Problems with iOS have also been registered before: https://gitmemory.com/issue/NativeScript/NativeScript/7085/494258757

Happily, I found out a native workaround. Not pretty, but works:

private layoutChangeListener = new android.view.View.OnLayoutChangeListener( {
  onLayoutChange (v: android.view.View, left:number, top:number, right:number, bottom:number, oldLeft:number, oldTop:number, oldRight:number, oldBottom:number):any {
      if (left != oldLeft || top != oldTop || right != oldRight || bottom != oldBottom) {
          console.log(`OnLayoutChangeListener left:${oldLeft}->${left}, top:${oldTop}->${top}, right:${oldRight}->${right}, bottom:${oldBottom}->${bottom}`);
          eval("_this._onLayoutChangeFn()");
      }
  }
});

initNativeView() {
  super.initNativeView();
  this.nativeView.addOnLayoutChangeListener(this.layoutChangeListener);
  ...
}
...
disposeNativeView() {
  this.nativeView.removeOnLayoutChangeListener(this.layoutChangeListener);

With that, CameraPlus._onLayoutChangeFn can be used to trigger an update on Fancycamera. Is that the best approach?

[Murilo-Perrone]

Found out the best approach which was to use onConfigurationChanged within Camera2.kt from FancyCamera to update the ImageCapture target rotation. I think that was the last fix needed for correct image capturing. 🥳

[Murilo-Perrone]

Fixed through an update in fancycamera dependency (was missing portrait adjustment for ImageCapture targets).