Camera view poor performance: delayed input

[nunoguedelha]

In case the currently used protocol (tcp) is suboptimal, we can choose a better protocol for sending image binary data. But first, we need to check if the delay is mostly due to the image data transfer or the image buffering and rendering at the client browser app.

Current Status

Analysis steps:

• [x] check if the delay is mostly due to the image data transfer or the image buffering on the client browser app.
• [x] Try other connection carriers (fast_tcp, http, mjpeg (mjpeg-over-http))
  • [x] fast_tcp
  • [x] http
  • [x] #107
• [x] Address buffering issue on client browser app if required. => workaround

Conclusion

TBD

[nunoguedelha]

In order to identify where the delay is being introduced, we write to an image port the image samples received by he client... https://github.com/ami-iit/yarp-openmct/blob/6a7258b90016ef93d1707bbe350c61c13478da5d/openmctStaticServer/realtime-telemetry-plugin.js#L17-L27

...and we compare two outputs:

• the output of a yarpview plugin device directly connected to the framegrabber port /icubSim/camLeftEye,

  yarpview --synch --keep-above --name /yarpview/imgdirect:i
  yarp connect /icubSim/camLeftEye /yarpview/imgdirect:i

• the output of a yarpview plugin device connected to the output port /yarpjs/camLeftEyeEcho:o.

  yarpview --synch --keep-above --name /yarpview/imgecho:i
  yarp connect /yarpjs/camLeftEyeEcho:o /yarpview/imgecho:i

We get an error on the telemetry server console:

[OPEN-MCT STATIC SERVER] stdout: Error: impossible to copy image

The problem is with the object type check in the yarp.Image class copy method:

yarp.Image = function Image(_image) {

    if(_image == undefined)
        var _image = _yarp.Image();

    // process.on('exit',function () {
    //     var b = _image;
    // });

    _image._copy = _image.copy;
    _image.copy = function (obj) {
        if(_image.getObjType == obj.getObjType)
            _image.copy(obj);
        else
            console.log('Error: impossible to copy image'); <=====
    }

Fixing...

[nunoguedelha]

Meanwhile I'm checking the fast_tcp, http and mjpeg (mjpeg-over-http) carriers documentation.

[nunoguedelha]

The problem is with the object type check in the yarp.Image class copy method:

• For starters there is a bug in the _image.copy wrapper method:

  _image._copy = _image.copy;
  _image.copy = function (obj) {
      if(_image.getObjType == obj.getObjType)
          _image.copy(obj);        <------------------- HERE
      else
          console.log('Error: impossible to copy image');
  }

  _image.copy(obj) should be _image._copy(obj) for calling the original copy method, otherwise we create a infinite loop recursive call.

• Sending an image sample through YARP from the browser is not properly implemented in the yarp.js package. The problem is bigger than just the object type check failure described earlier. I've opened an issue on yarp.js repo: https://github.com/robotology/yarp.js/issues/38 .

For that reason I´ve decided to use other approaches for checking the cause of the end-to-end transmission delay:

• [ ] Render the received samples in an HTML canvas as done in the yarp.js example https://github.com/robotology/yarp.js/blob/master/examples/stream_video/index.html.
• [ ] Insert a timestamp in the image sample and check wether the difference with a local timestamp increases or stays constant.

[nunoguedelha]

WIP: Render the received samples in an HTML canvas as done in the yarp.js example https://github.com/robotology/yarp.js/blob/master/examples/stream_video/index.html

• I first tried to do it by directly calling video_stream_element.getContext('2d').drawImage(canvasImg,0,0) from https://github.com/ami-iit/yarp-openmct/blob/6a7258b90016ef93d1707bbe350c61c13478da5d/openmctStaticServer/realtime-telemetry-plugin.js#L5 . This fails as the canvas is intermittently updated.
• I then tried skipping all Open-MCT plugins installations, and instead, just displaying the canvas. => Scrolling line [´mode --line´]: unlike what we observe on the YARP viewer, when the line reaches the bottom of the canvas, it doesn´t instantly appear at the top. Instead, it remains invisible for a while, then appears at the top and scrolls faster than the YARP viewer. => Boucing ball [´mode --ball´]: doesn´t appear at all.
• For checking if there was an occasional issue with my setup, I went back to the good old yarp.js stream video example. It also fails. Checking the setup... ==> Actually this is a yarp.js issue. Opened https://github.com/robotology/yarp.js/issues/39 .

[nunoguedelha]

Try other connection carriers

We try here other connection carriers (fast_tcp, http, mjpeg (mjpeg-over-http)).

• http ==> YARP ports from your browser:
  • http://192.168.1.70:10003/* gives the description of the port

    This is /icubSim/camLeftEye at tcp://192.168.1.70:10003/
    There is an output connection from /icubSim/camLeftEye to web using http
    There is an input connection from web to /icubSim/camLeftEye using http

  • http://192.168.1.70:10003/r reads the port in text mode (no output in our case)
  • http://192.168.1.70:10003/data=stream => NOT WORKING, I have to check my YARP version.
• fast_tcp ==> NO IMPROVEMENT. This happens if the data flow is consumer-limited. So this increases our belief in the issue being on the client side.
• mjpeg ==> Requires rebuilding. Having some build issues. Will add details later.

[nunoguedelha]

Render the received samples in an HTML canvas as done in the yarp.js "stream_video" example (Final)

• For checking if there was an occasional issue with my setup, I went back to the good old yarp.js stream video example. It also fails. Checking the setup... ==> Actually this is a yarp.js issue. Opened

This was a scaling issue in the canvas drawImage method parameters (refer to fix in the yarp.js video stream example).

• I then tried skipping all Open-MCT plugins installations, and instead, just displaying the canvas.

After this fix...

• The test animation scrolling line is working fine: the scrolling and transition from bottom to top is uninterrupted and smooth.
• There is no delay compared to the reference yarpview visualization.

https://github.com/ami-iit/yarp-openmct/commit/f00ef06a2a1327c33af56167f7c4ac2b828b4875

https://user-images.githubusercontent.com/6848872/164045674-63ef1a0a-0dde-42cb-a8cb-133c89a2bf79.mp4

[nunoguedelha]

Try other connection carriers

• mjpeg ==> Requires rebuilding. Having some build issues. Will add details later.

https://docs.drdanz.it/git-master/carrier_config.html#carrier_config_mjpeg

YARP CMake configuration (configured and built through superbuild)

YARP_COMPILE_CARRIER_PLUGINS = ON
ENABLE_yarpcar_mjpeg_carrier = ON
MJPEG_AUTOCOMPRESS = ON

After running a test image FakeFrameGrabbertest...

yarpdev --file fakeFrameGrabber_basic.ini --mode line

with fakeFrameGrabber_basic.ini defined as

device fakeFrameGrabber
name /icubSim/camLeftEye
width 640
height 480
period 10

... and connecting to yarpview ...

yarp connect /icubSim/camLeftEye /yarpview/img:i mjpeg

...we get the following error on the console

Wrong JPEG library version: library is 90, caller expects 80

Trying to read the port from a browser (Firefox) by querying http://192.168.1.70:10003/?action=stream also fails.

I tried inverting src/dest parameters, since mjpeg is a pull protocol, but it didn't help.

[nunoguedelha]

I tried inverting src/dest parameters, since mjpeg is a pull protocol, but it didn't help.

Disregard this comment. The correct order is still yarp connect <out-port> <in-port> mjpeg

yarp connect /icubSim/camLeftEye /yarpview/img:i mjpeg

and because mjpeg is a "pull" protocol we get the console [INFO] message with the "inverted" order ...

[INFO] |yarp.os.Network| Success: Added connection from /yarpview/img:i to mjpeg://icubSim/camLeftEye

[nunoguedelha]

Some references for running or configuring MJPEG protocol: https://github.com/robotology/community/discussions/32 https://docs.drdanz.it/git-master/carrier_config.html#carrier_config_http

Some references on the issue "Wrong JPEG library version: library is 90, caller expects 80" https://github.com/openMVG/openMVG/issues/1349

[nunoguedelha]

Try other connection carriers

dropping the analysis for now, for focusing on "Address buffering issue on client browser app if required".

[nunoguedelha]

Buffering issue on client browser app if required.

Quickly summarizing here the observed problem and analysis with few details. A workaround will follow, which improves a bit the performance but not enough. A more thorough analysis shall be done for a permanent better fix.

On the Visualization Window Boundaries

• The received frames are buffered in a dynamic buffer (https://github.com/nasa/openmct/blob/d30c4fcb5371aeffbb9b4464a43d3b701da3eae1/src/plugins/imagery/mixins/imageryData.js#L190).
• The frames are rendered in the main canvas container and thumbnails if their timestamp falls within the time boundaries of the sliding visualization window.
• The past frames seem never to be discarded (to be checked again).
• The time boundaries are updated w.r.t. the current timestamps

- => https://github.com/nasa/openmct/blob/d30c4fcb5371aeffbb9b4464a43d3b701da3eae1/src/plugins/utcTimeSystem/LocalClock.js#L58 ```js tick() { const now = Date.now(); super.tick(now); <-------- this.timeoutHandle = setTimeout(this.tick.bind(this), this.period); } ``` - => https://github.com/nasa/openmct/blob/d30c4fcb5371aeffbb9b4464a43d3b701da3eae1/src/utils/clock/DefaultClock.js#L44 ```js tick(tickValue) { this.emit("tick", tickValue); <-------- this.lastTick = tickValue; } ``` - => https://github.com/nasa/openmct/blob/d30c4fcb5371aeffbb9b4464a43d3b701da3eae1/src/api/time/GlobalTimeContext.js#L69 ```js tick(timestamp) { super.tick.call(this, ...arguments); <-------- // If a bounds change results in a TOI outside of the current // bounds, unset it if (this.toi < this.boundsVal.start || this.toi > this.boundsVal.end) { this.timeOfInterest(undefined); } } ``` - => https://github.com/nasa/openmct/blob/d30c4fcb5371aeffbb9b4464a43d3b701da3eae1/src/api/time/TimeContext.js#L362 ```js tick(timestamp) { if (!this.activeClock) { return; } const newBounds = { start: timestamp + this.offsets.start, end: timestamp + this.offsets.end }; this.boundsVal = newBounds; <-------- this.emit('bounds', this.boundsVal, true); } ```

• As the rendering flow slows down and stutters, we observe that the timestamp of the received frames is getting closer and closer to the upper boundary (boundary ahead of the current local timestamp).

Possible Root Causes

• The image frames are being pushed into an ever growing buffer, which is probably causing the delayed rendering (e.g. due to a sub-optimal management of the buffer size and the refresh of the time strip thumbnails below the image canvas) => Indeed, reducing the buffer size to 1 or 2 samples reduces the stutter.^*

(*) As I mentioned in the call, I had already tried reducing the sliding window depth to just a few samples through the UI, but that wasn't affecting the performance. Only changing the buffering procedure in the code directly had an effect: https://github.com/nasa/openmct/blob/d30c4fcb5371aeffbb9b4464a43d3b701da3eae1/src/plugins/imagery/mixins/imageryData.js#L190

        subscribe() {
            this.unsubscribe = this.openmct.telemetry
                .subscribe(this.domainObject, (datum) => {
                    let parsedTimestamp = this.parseTime(datum);
                    let bounds = this.timeContext.bounds();

                    if (parsedTimestamp >= bounds.start && parsedTimestamp <= bounds.end) {
                        let image = this.normalizeDatum(datum);
                        if (image) {
                            this.imageHistory.push(image);
                            if (this.imageHistory.length > 2) { <-----------
                               this.imageHistory.shift();
                            }
                        }
                    }
                });
        },

• Without this workaround, and despite the slowdown of the rendering, every-time the boundaries are updated w.r.t. the local clock, the rendering should catch up with the latest received frames, but we are not observing this. Instead, the displayed image lags way behind the last received one (e.g. scrolling line lagging behind the reference one from the yarpview renderer). This might be caused by a local clock running slower than the Yarp clock. => To Be Analyzed.

[nunoguedelha]

@traversaro , the workaround/fixes are to be done in the openmct module, so we need to fork the repo https://github.com/nasa/openmct.git as we discussed last time. I was doing this in a personal fork on my Github account, but we should a priori fork the openmct repo either on robotology or ami organisations. I think on robotology it would be better as it is widely accessible. What do you think?

But in that case I don't have the rights...

[traversaro]

Done: https://github.com/ami-iit/openmct . You have admin rights.