Bad algorithms for capture data from streams and missing frames severely

[YowFung]

Hi @jaumard

I think the core algorithm is problematic. It does not correctly capture a frame of a full MJPEG image.

This your code :

var chunks = <int>[];
_subscription = response.stream.listen((data) async {
  if (chunks.isEmpty) {
    final startIndex = data.indexOf(_trigger);
    if (startIndex >= 0 &&
        startIndex + 1 < data.length &&
        data[startIndex + 1] == _soi) {
      final slicedData = data.sublist(startIndex, data.length);
      chunks.addAll(slicedData);
    }
  } else {
    final startIndex = data.lastIndexOf(_trigger);
    if (startIndex + 1 < data.length && data[startIndex + 1] == _eoi) {
      final slicedData = data.sublist(0, startIndex + 2);
      chunks.addAll(slicedData);
      final imageMemory = MemoryImage(Uint8List.fromList(chunks));
      await precacheImage(imageMemory, context);
      errorState.value = null;
      image.value = imageMemory;
      chunks = <int>[];
      if (!isLive) {
        dispose();
      }
    } else {
      chunks.addAll(data);
    }
  }
}, onError: (err) {
  errorState.value = err;
  image.value = null;
  dispose();
}, cancelOnError: true);

The following are the reasons for the failure to detect correctly :

1. During soi detection, the data flow will not be properly detected when it is the following. Because you only detected the FF sign once by indexOf. But it's actually possible after that.

   ... FF ... FF D8 ...

2. During eoi detection, the data flow will not be properly detected when it is the following. Because you only detected the FF sign once by lastIndexOf. But it's actually possible before that.

   ... FF D9 ... FF ...

The effect for me :

When I discovered this problem, I ran a frame-rate test on the MJPEG video stream. It's only 7-8 fps, but actually is 15. Half of it is missing.

I/flutter (18349): --------- fps: 8.286008486364173
I/flutter (18349): --------- fps: 7.617952375151854
I/flutter (18349): --------- fps: 8.410097196334307
I/flutter (18349): --------- fps: 8.370997940985635
I/flutter (18349): --------- fps: 7.984312422951385
I/flutter (18349): --------- fps: 7.878728095954084
I/flutter (18349): --------- fps: 8.468421594610561
......

[jaumard]

Hello, for the FPS did you try in debug or release mode ? Because it change things a lot.

I'll try to dig into that asap on my side :)

[YowFung]

It happens in both modes. By the way, it's possible that the FF D9 and FF D8 in the same sliced data. FF D8 is probably next to FF D9.

[jaumard]

I'll try a fix and push it to a branch, please let me know how it goes

[jaumard]

@YowFung can you try the branch bugfix/missingFrame please ? should be better

[jaumard]

Let me know :)

[YowFung]

Oh, sorry. I'll try it tomorrow because I have no environment now. According your code I think that it will appear the following problems, but I'm not sure now.

1. It cannot find eoi timely when the data variable contains the soi and eoi at the same time.
2. You will lose the rest of data if the soi is next to eoi and them in the same data.
3. The efficiency of finding eoi is very low because it traverses the lookup from scratch every time. sorry!!

[jaumard]

You're right but I didn't 1 and 2 were actual use case we can have :) I'll try to have a look before tomorrow then. Thanks!

[YowFung]

Hi @jaumard . It's just what I thought: The frame rate is sometimes 8-9 fps, and sometimes 13-14, but really is about 15.

I/flutter (21549): ----------------- fps: 13.715964861892578
I/flutter (21549): ----------------- fps: 9.580856681881068
I/flutter (21549): ----------------- fps: 14.157105075874297
I/flutter (21549): ----------------- fps: 14.04556916368327
I/flutter (21549): ----------------- fps: 8.339926740415526
I/flutter (21549): ----------------- fps: 13.767103561109144
I/flutter (21549): ----------------- fps: 9.771106934505736
I/flutter (21549): ----------------- fps: 13.777750283511656
I/flutter (21549): ----------------- fps: 13.964752406336311
I/flutter (21549): ----------------- fps: 8.494923051712759
I/flutter (21549): ----------------- fps: 14.230232641535316
I/flutter (21549): ----------------- fps: 14.143339349909892
I/flutter (21549): ----------------- fps: 7.195245726995396
I/flutter (21549): ----------------- fps: 14.288863959584233
I/flutter (21549): ----------------- fps: 9.116663665764877
......

[jaumard]

Yes I've search a bit and for now not sure what's the best way to implement this algo. At least there is an improvement already lol I'll give it another try today and keep you updated !

[YowFung]

I have designed an algorithm that can perfectly solve this problem, but due to bugs elsewhere, I cannot test it now, so I cannot verify whether this algorithm is effective.

class MjpegStream extends ChangeNotifier
{
  ......

  final String url;
  final String method;
  final Map<String, String> headers;
  final Duration timeout;

  bool _running = false;
  Uint8List _imageData;
  String _errMsg;

  StreamSubscription _subscription;
  Client _httpClient;
  bool _mutexLock = false;
  List<int> _buffer = [];
  int _startIndex = -1;
  int _endIndex = 2;

  Future<void> start() async {
    try {
      this._running = true;
      this.notifyListeners();

      final request = Request(this.method, Uri.parse(this.url));
      request.headers.addAll(this.headers ?? Map<String, String>());
      this._httpClient = new Client();
      final response = await this._httpClient.send(request).timeout(this.timeout);

      if (response.statusCode >= 200 && response.statusCode < 300) {
        this._mutexLock = false;
        this._startIndex = -1;
        this._endIndex = 2;
        this._buffer.clear();

        this._subscription = response.stream.listen((data) async { 
          while(this._mutexLock) await Future.delayed(Duration(microseconds: 10));
          this._mutexLock = true;
          this._buffer.addAll(data);

          if (this._startIndex == -1) {
            var index = 0;
            var finished = false;
            do {
              index = this._buffer.indexOf(MjpegStream._trigger, index);
              finished = index == -1 || index+1 >= this._buffer.length;
              if (finished == false && this._buffer[index+1] == MjpegStream._soi) {
                this._startIndex = index;
                this._endIndex = 2;
                this._buffer = this._buffer.sublist(index);
                finished = true;
              }
              else
                index += 1;
            } while (finished == false);
          }

          else if (this._buffer.length > 2) {
            var finished = false;
            do {
              this._endIndex = this._buffer.indexOf(MjpegStream._trigger, this._endIndex);
              finished = this._endIndex == -1 || this._endIndex+1 >= this._buffer.length;
              if (finished == false && this._buffer[this._endIndex+1] == MjpegStream._eoi) {
                final frameData = this._buffer.sublist(0, this._endIndex+2);
                this._imageData = Uint8List.fromList(frameData);
                this._errMsg = null;
                this._buffer = this._buffer.sublist(this._endIndex+2);
                this._startIndex = -1;
                this._endIndex = 2;
                finished = true;
                this.notifyListeners();
              }
              else 
                this._endIndex += 1;
            } while (finished == false);
          }

          this._mutexLock = false;

        }, onError: (err) {
          this._pauseWithError(err.toString());
        }, cancelOnError: true);
      }

      else 
        this._pauseWithError("Stream returned ${response.statusCode} status.");
    }

    catch(error) {
      this._pauseWithError(error.toString());
    }
  }

  ......
}

[jaumard]

Not sure I understand your code lool too obscur to me compare to what I already have :/ I have tried something really simple to see if it's better. But for some reason I still have crash on some frames so there is a bug somewhere but can't find it for now. I'll keep searching but let me know if it improve a bit on your side

[jaumard]

any luck testing @YowFung ? I'm still not able to find my bug, might need some help if you can :)

[jaumard]

I've update a bit, it's better now but I still get some errors, @YowFung any feedback ?

[jaumard]

Should be fixed in 1.3.0