Preview image sometimes distorted after orientation change

[3ph]

I've got a button in my app which would force orientation change (from portrait to landscape and back). Most of the time this works fine and the image is updated correctly with the orientation change. Occasionally though (and there doesn't seem to be a consistent way to reproduce this) the image doesn't rotate and gets distorted.

I believe this might be iOS library issue as the orientation is correctly detected in the CameraPreview widget. Interestingly the image gets fixes not only when the device is physically rotated but also when I lay it flat (and there is no physical orientation change, I'm listening to the physical orientation changes using device_orientation package).

[ThibaultBee]

Hi Tom,

Could you share your code to have a look on this? As I undertand it, you have 2 issues:

• preview is distorded on orientation changed (random). Do you have a screenshot?
• preview is stuck on flat (reproductible). In this case, do you have any idea what orientation NativeDeviceOrientationReader has detected?

[3ph]

Hi,

Here is the code for listening to device orientation and forcing it (after device orientation changes and also when user taps button).

  @override
  void initState() {
    super.initState();
    // this comes from device_orientation package
    _orientationSubscription = deviceOrientation$.listen((orientation) {
      _deviceOrientation = orientation.fixed;
      print('Device orientation: $_deviceOrientation');
      _forceOrientation();
    });
}

  Future<void> _forceOrientation({bool? shouldLockPortrait}) async {
    if (shouldLockPortrait != null) {
      _shouldLockPortrait = shouldLockPortrait;
      // force orientation button was pressed
      // check if we are in correct device orientation
      if ((shouldLockPortrait && _deviceOrientation.isPortrait) ||
          !(shouldLockPortrait || _deviceOrientation.isPortrait)) {
        await SystemChrome.setPreferredOrientations([
          _deviceOrientation,
        ]);
        print('Forcing orientation1: $_deviceOrientation');
      } else {
        // use defaults
        await SystemChrome.setPreferredOrientations([
          shouldLockPortrait
              ? DeviceOrientation.portraitUp
              : DeviceOrientation.landscapeRight,
        ]);
        print(
            'Forcing orientation2: ${shouldLockPortrait ? DeviceOrientation.portraitUp : DeviceOrientation.landscapeRight}');
      }
    } else {
      // native orientation changed, check if we are in correct lock orientation
      // otherwise ignore
      if ((_shouldLockPortrait && _deviceOrientation.isPortrait) ||
          !(_shouldLockPortrait || _deviceOrientation.isPortrait)) {
        await SystemChrome.setPreferredOrientations([
          _deviceOrientation,
        ]);
        print('Forcing orientation3: $_deviceOrientation');
      }
    }
  }

The CameraPreview widget is called just with controller parameter.

I've only got an issue with the distorted image, not when it's flat (that would actually fix the first issue). I've put some printout in the CameraPreview widget on _isLandscape and it's matching the forced orientation.

Screenshot is here.

[ThibaultBee]

I could not find any relative issue in HaishinKit. Could you fork this repo and add a button to set preview orientation? Is your live stream correctly oriented when you set the orientation?

[3ph]

Yeah, I could not find anything either. Not sure if I understand what do you mean by adding a button to set the preview orientation?

When this issue happens the distortion is caused by the wrong aspect ratio (and obviously it's orientated wrong). So for example I force landscape orientation (when the device is landscape orientated). The UI updates correctly, the preview detects the correct orientation as well and sets the aspect ratio for landscape. But the preview stream is still coming in as portrait (and the image gets distorted because of the aspect ratio set for landscape). When I start streaming the live stream is recorded as a portrait (with no distortion obviously).

[3ph]

@ThibaultBee have you got any suggestions?

[ThibaultBee]

Hi, Have you tried with v1.0.6? I just remember I made changed on the view for iOS. Instead of using a MTHKView, we directly implement a NetStreamDrawable. If that does not fix this issue, I don't have any clue!

[3ph]

Yes, 1.0.6, unfortunately, doesn't seem to fix it :(

[3ph]

I've also tried to replace the CameraPreview from apivideo package with the one from camera package and it doesn't seem to be affected by this issue.

[ThibaultBee]

hum, if there is an issue with the camera preview orientation, it migh be with NativeDeviceOrientation. This is the only part that deal with the preview orientation. The camera package does not use NativeDeviceOrientation.

Could you fork this repository and add your code to listen to device orientation and forcing it in the example? Please

[3ph]

But the orientation is detected correctly by the CameraPreview widget (from both packages), it's the stream which is wrong.

[ThibaultBee]

But there isn't anything about the orientation in the stream:

• See HK (drawable): https://github.com/shogo4405/HaishinKit.swift/blob/54694fe638e30a7fa0b3b72393871097cfbe6c57/Sources/Media/AVVideoIOUnit.swift#L386
• See FlutterTexture: https://github.com/apivideo/api.video-flutter-live-stream/blob/main/ios/Classes/FlutterTexture.swift

Maybe in a specific case, the NativeDeviceOrientation have an odd orientation detection.

[ThibaultBee]

oh, yes, you are right, there is nothing in the CameraPreview on orientation for iOS: https://github.com/apivideo/api.video-flutter-live-stream/blob/947a7074e89a966f5af816490ecc1501ab871954/lib/apivideo_live_stream.dart#L188

[ThibaultBee]

So the issue might be somewhere here: https://github.com/apivideo/api.video-ios-live-stream/blob/2aab29979327ec71ce6710f74d439fe12b28ebcb/Sources/ApiVideoLiveStream.swift#L286

[ThibaultBee]

Hi, Somehow, in the native part orientationDidChange is not called sometimes. Maybe when you unlock the orientation? Do you properly reset the orientation when you unlock the orientation?

[3ph]

I'm just using SystemChrome.setPreferredOrientations([...]) to set it to preferred orientation. I don't unlock it on this page - I mean the issue happens when I keep switching between landscape vs portrait.

[ThibaultBee]

I am sorry to say this but I am kind of lost here. So the issue happens on when you keep switching between landscape vs portrait? Have you lock the orientation in this case?

Could you provide an example I can directly run on my phone and explain step by step how to reproduce this issue?

[3ph]

Yes, if I keep switching between landscape and portrait (using the setPreferredOrientations) while physically rotating the device the issue would sooner or later manifest. I'm using the code I pasted above but will create complete gist for you so you can test yourself.

[3ph]

Ok, I've created the sample app here. There is a readme describing the steps and also screenshot as well.

[ThibaultBee]

Ok to reproduce this issue:

Initial step:

• Application is lock in portrait and device is in portrait
• Turn your device to landscape
• Lock the application in landscape

The preview is rotated a quarter more than it should be. In this case, orientationDidChange observer is not called.

[ThibaultBee]

I spent the day trying to understand this issue but unfortunately I don't understand where it comes from. (HK? Live stream fltter native part? Flutter?). I don't really understand how the preview of the camera package works neither...

[3ph]

Hi Thibault, thanks a lot for putting so much effort into this issue. What do you suggest now?

[3ph]

Hi Thibault, I have opened an issue with HaishinKit but probably have not enough information to answer him. Issue here https://github.com/shogo4405/HaishinKit.swift/issues/1076. Thanks

[ThibaultBee]

According to the issue on HaishinKit, it seems to be an issue in Flutter. There is nothing we can do on our side, maybe one day it will be fixed by a new flutter version.