Apparence-io / CamerAwesome

📸 Embedding a camera experience within your own app shouldn't be that hard. A flutter plugin to integrate awesome Android / iOS camera experience.
https://ApparenceKit.dev
MIT License
905 stars 199 forks source link

webrtc p2p connection video freezes with CamerAwesome #390

Open azazadev opened 9 months ago

azazadev commented 9 months ago

in p2p connection when I'm using RTCVideoView for both local and remote user it's working fine.

Now I have the scenario when I want to replace the RTCVideoView local widget by the CameraAwesomeBuilder.previewOnly ( needed for onImageForAnalysis use case ), the p2p connection is established but local video is freezes

any idea why CameraAwesomeBuilder is freezes ?

NOTE : probably this is not related to this amazing package but any help is appreciated

This is a full example when we can reproduce issue easy :

How to reproduce :

  1. run below code on Device 1
  2. run below code on Device 2
  3. copy the connection ID of Device 1 on the TextField of Device 2
  4. click on connect

Observed : connection is established and device 2 is ok with local / remote video but device 1 video freezes

import 'dart:math';

import 'package:camera/camera.dart';
import 'package:camerawesome/camerawesome_plugin.dart';
import 'package:flutter/material.dart';
import 'package:flutter_webrtc/flutter_webrtc.dart' as flutter_webrtc;
import 'package:loading_animation_widget/loading_animation_widget.dart';
import 'package:peerdart/peerdart.dart';

Future<void> main() async {
  try {
    WidgetsFlutterBinding.ensureInitialized();
  } on CameraException catch (e) {}
  runApp(MaterialApp(
    home: WebRTCP2PDemo2(),
  ));
}

class WebRTCP2PDemo2 extends StatefulWidget {
  const WebRTCP2PDemo2({Key? key}) : super(key: key);

  @override
  _WebRTCP2PDemo2State createState() => _WebRTCP2PDemo2State();
}

class _WebRTCP2PDemo2State extends State<WebRTCP2PDemo2> {
  final TextEditingController _controller = TextEditingController();
  late Peer peer = Peer(options: PeerOptions(debug: LogLevel.All), id: GeneratePeerId());
  final _localRenderer = flutter_webrtc.RTCVideoRenderer();
  final _remoteRenderer = flutter_webrtc.RTCVideoRenderer();
  String? peerId;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
          child: Column(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          SizedBox(
            height: 50,
          ),
          const Text(
            'Connection ID:',
          ),
          SelectableText(
            peerId ?? '',
            style: TextStyle(fontSize: 48),
          ),
          TextField(
            textAlign: TextAlign.center,
            keyboardType: TextInputType.number,
            controller: _controller,
            style: TextStyle(fontSize: 48),
          ),
          ElevatedButton(onPressed: _connect, child: const Text("connect")),
          Expanded(
            child: Padding(
              padding: const EdgeInsets.all(8.0),
              child: Container(
                decoration: BoxDecoration(
                  border: Border.all(width: 3.0, color: Colors.black38),
                  color: Colors.transparent,
                  borderRadius: BorderRadius.all(Radius.circular(5.0)),
                ),
                child: SizedBox(
                  height: MediaQuery.of(context).size.height / 4,
                  width: MediaQuery.of(context).size.width,
                  child: flutter_webrtc.RTCVideoView(
                    _remoteRenderer,
                    placeholderBuilder: (ctx) {
                      return CircularProgressIndicator();
                    },
                  ),
                ),
              ),
            ),
          ),
          Expanded(
            child: Padding(
              padding: const EdgeInsets.all(8.0),
              child: Container(
                decoration: BoxDecoration(
                  border: Border.all(width: 3.0, color: Colors.black38),
                  color: Colors.transparent,
                  borderRadius: BorderRadius.all(Radius.circular(5.0)),
                ),
                child: SizedBox(
                  height: MediaQuery.of(context).size.height / 3,
                  width: MediaQuery.of(context).size.width,
                  child: CameraAwesomeBuilder.previewOnly(
                    sensorConfig: SensorConfig.single(
                      sensor: Sensor.position(SensorPosition.front),
                      aspectRatio: CameraAspectRatios.ratio_1_1,
                    ),
                    previewFit: CameraPreviewFit.cover,
                    progressIndicator: Container(
                      color: Colors.transparent,
                      child: Center(
                        child: LoadingAnimationWidget.halfTriangleDot(
                          color: Colors.white,
                          size: MediaQuery.of(context).size.width / 5,
                        ),
                      ),
                    ),
                    builder: (cameraModeState, previewSize, previewRect) {
                      return AwesomeCameraLayout(
                        state: cameraModeState,
                        topActions: SizedBox.shrink(),
                        middleContent: SizedBox.shrink(),
                        bottomActions: SizedBox.shrink(),
                      );
                    },
                    onImageForAnalysis: (img) => _processImage(img),
                    imageAnalysisConfig: AnalysisConfig(
                      androidOptions: const AndroidAnalysisOptions.nv21(width: 640),
                      cupertinoOptions: const CupertinoAnalysisOptions.bgra8888(),
                      maxFramesPerSecond: 10,
                    ),
                  ),
                ),
              ),
            ),
          ),
        ],
      )),
    );
  }

  Future _processImage(AnalysisImage decodeImage) async {}

  String GeneratePeerId() {
    Random random = Random();
    int min = 100000;
    int max = 999999;
    int randomNumber = min + random.nextInt(max - min);
    return randomNumber.toString();
  }

  @override
  void initState() {
    super.initState();
    peer.on("open").listen((id) {
      setState(() {
        peerId = peer.id;
      });
    });
    _remoteRenderer.initialize();
    _localRenderer.initialize();
    peer.on<MediaConnection>("call").listen(
      (call) async {
        final mediaStream = await flutter_webrtc.navigator.mediaDevices.getUserMedia({
          'video': {
            'facingMode': 'user',
          },
          "audio": false
        });
        call.answer(mediaStream);
        call.on("close").listen((event) {
          setState(() {});
        });

        call.on<flutter_webrtc.MediaStream>("stream").listen(
          (event) {
            _localRenderer.srcObject = mediaStream;
            _remoteRenderer.srcObject = event;
            setState(() {});
          },
        );
      },
    );
  }

  void _connect() async {
    FocusManager.instance.primaryFocus?.unfocus();
    final mediaStream = await flutter_webrtc.navigator.mediaDevices.getUserMedia({
      'video': {
        'facingMode': 'user',
      },
      "audio": false
    });
    final conn = peer.call(_controller.text, mediaStream);

    conn.on("close").listen((event) {
      setState(() {});
    });

    conn.on<flutter_webrtc.MediaStream>("stream").listen((event) {
      _remoteRenderer.srcObject = event;
      _localRenderer.srcObject = mediaStream;
      setState(() {});
    });
  }

  @override
  void dispose() {
    peer.dispose();
    _remoteRenderer.dispose();
    _localRenderer.dispose();
    super.dispose();
  }
}

(base) ➜ ~ flutter doctor Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 3.13.1, on macOS 13.1 22C65 darwin-arm64, locale en-FR) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3) [✓] Xcode - develop for iOS and macOS (Xcode 14.2) [✓] Chrome - develop for the web [✓] Android Studio (version 4.0) [✓] Android Studio (version 2021.1) [✓] IntelliJ IDEA Community Edition (version 2022.3.1) [✓] VS Code (version 1.79.2) [✓] Connected device (3 available) [✓] Network resources

• No issues found!

g-apparence commented 9 months ago

That's interesting. P2P vidéo was something I had in mind for a long time.

I am a bit really busy this week but I'll try to investigate.

At first sight I would say that the video source cannot be listened from camerAwesome and webrtc. They should share the same Stream. But I'll try to confirm that.