Problems with playback of network stream

[Ligverd]

When playing a network stream live video through udpxy or just file posted on the web server, on some devices the image and the sound starts to stutter, to fail, to fall down. On the console when a bunch of errors in the decoding of the stream. Tries to decode up to the time when the network buffer stream is not full, i.e., there is minimal data to decode. With playback of files from flash memory no problem. Need to pause before decoding, i.e., to wait for a certain fullness of the read buffer. Calculate the buffer size depending on the bitrate. Help or tell me where to correct this.

android-ndk r10d qt 5.4.0 qtav master openal 1.16.0 ffmpeg 2.5.3

[wang-bin]

Not sure. But I'm going to add this feature, start to decode only if there are enough packets. I plan to finish it in 2 weeks.

[Ligverd]

good!

I understand that this is not exactly it, but because of the increase of the output audio buffer and async with some artifacts at the beginning of the play and in the future with a little lag in the video from the audio, I was able to achieve a normal audio and video playback.

in AudioOutput_p.h

// chunk const int kBufferSize = 1024*128; //4;

Looking mostly to no avail. AVDemuxer.cpp AVInput.cpp

by the way without openal sound there is in principle acceptable, there is some small ripples similar to clipping. I.e. direct openSLES on Android works.

[wang-bin]

You can call player->audio()->setBufferSize(1024*128). What's the result if change to 1024*128?

AudioOutput is not completed yet. Only openal and portaudio works fine now. Improving audio is also in my plan in the following 2 weeks.

[Ligverd]

Increasing the buffer I was thinking of the following: as synchronization goes from the audio , the program will wait for the full population of a single audio buffer , thereby slow down the playback. I mentioned that this is not exactly what you need to do is just one very bad ways to achieve an acceptable quality playback. So you don't have to pay attention to it. The algorithm playback through OpenSLES explicitly contains an error. It is very tempting to get rid of dependence OpenAL

[wang-bin]

The algorithm playback through OpenSLES explicitly contains an error. What error?

[Ligverd]

I deal with bad playback using OpenSL Guessing that switching buffers occurs earlier time, i.e. the buffer is not yet finish and callback already switched to the new buffer. In the found sources on the Internet in callback function checks the readiness to switch.

for example saw this review. if( event & SL_PLAYEVENT_HEADATEND ) is_done_buffer = true;

Again made an experiment. took the pieces caught in buffer function AudioOutputOpenSL::write bite size is 4096 bytes put together all the pieces but cut off from each tail of 200 bytes turned out one to one, recorded the sound through line looked in Audacity on the waveform, fits, too.

Maybe this data will help you to understand what it is, I unfortunately don't poluhilos to understand.

[wang-bin]

thanks for your experiment. sounds reasonable. i have searched the event you mentioned and will havd a try

[Ligverd]

Add more. In OpenAL through the network with a standard buffer size 1024 * 4 and 8 buff_count sound and visible twitches a little out of sync audio and video. When installing Number of buffers 16 and former buffer size 1024 * 4

m_player->audio()->setBackends(QStringList()<<"OpenAL"); m_player->audio()->setBufferSize(1024*4); m_player->audio()->setBufferCount(16);

twitching sound stop BUT out of synch audio and video reaches 1 second, which is very noticeable. Ie Solving this problem can be safely used OpenAL

[Ligverd]

There will be a lot of text. :)

As mentioned above, by increasing or BufferCount BufferSize wrenching sound disappears but there is a lag of a sound from video. The Internet has found that OpenAL gives a lag of 0.5 s, but the proposed methods did not help to shift the audio. Don't know this is the case or not, but if you increase the buffer increases the lag. when BufferCount: 16 200ms lag when BufferCount: 32 400ms lag

I stopped to find out why this is happening and went the other way. I wrote the shift mechanism video to audio avShift

By the way there is such a possibility in some players, can come in handy in QtAV.

All decorated by analogy, i.e., in the AVPlayer class and also made in QML. The only thing that confuses me, if I did something that made setBufferCount in QML I made the restrictions so as not to compromise the integrity , say so to me this functionality is needed.

For the "purity" of the code please do not scold, would be very grateful if you could look this patch might make some edits and include it in the main branch.

diff was done to commit 428b8fdec0303f62efd85f8a76fff10519238e33 I saw what you did Comit the changes in VideoThread but I haven't tried them.

My working settings BufferCount: 16 avShift: 0.200

diff --git a/qml/QmlAV/QmlAVPlayer.h b/qml/QmlAV/QmlAVPlayer.h
index 9dc9517..39119d4 100644
--- a/qml/QmlAV/QmlAVPlayer.h
+++ b/qml/QmlAV/QmlAVPlayer.h
@@ -50,6 +50,8 @@ class QmlAVPlayer : public QObject, public QQmlParserStatus
     Q_PROPERTY(bool autoPlay READ autoPlay WRITE setAutoPlay NOTIFY autoPlayChanged)
     Q_PROPERTY(bool autoLoad READ isAutoLoad WRITE setAutoLoad NOTIFY autoLoadChanged)
     Q_PROPERTY(qreal playbackRate READ playbackRate WRITE setPlaybackRate NOTIFY playbackRateChanged)
+    Q_PROPERTY(qreal shift READ shift WRITE setShift NOTIFY shiftChanged)
+    Q_PROPERTY(int aoBufferCount READ aoBufferCount WRITE setAOBufferCount NOTIFY aoBufferCountChanged)
     Q_PROPERTY(QUrl source READ source WRITE setSource NOTIFY sourceChanged)
     Q_PROPERTY(int loops READ loopCount WRITE setLoopCount NOTIFY loopCountChanged)
     Q_PROPERTY(qreal bufferProgress READ bufferProgress NOTIFY bufferProgressChanged)
@@ -148,6 +150,10 @@ public:
     void setPlaybackState(PlaybackState playbackState);
     qreal playbackRate() const;
     void setPlaybackRate(qreal s);
+    qreal shift() const;
+    void setShift(qreal s);
+    int aoBufferCount() const;
+    void setAOBufferCount(int s);
     Q_INVOKABLE void play(const QUrl& url);
     AVPlayer *player();

@@ -220,6 +226,8 @@ Q_SIGNALS:
     void playbackStateChanged();
     void autoPlayChanged();
     void playbackRateChanged();
+    void shiftChanged();
+    void aoBufferCountChanged();
     void paused();
     void stopped();
     void playing();
diff --git a/qml/QmlAVPlayer.cpp b/qml/QmlAVPlayer.cpp
index dcc0b48..77c1c48 100644
--- a/qml/QmlAVPlayer.cpp
+++ b/qml/QmlAVPlayer.cpp
@@ -520,6 +520,22 @@ void QmlAVPlayer::setPlaybackRate(qreal s)
     emit playbackRateChanged();
 }

+qreal QmlAVPlayer::shift() const
+{
+    if (mpPlayer)
+        return mpPlayer->shift();
+    else
+        return 0;
+}
+
+void QmlAVPlayer::setShift(qreal s)
+{
+    if (mpPlayer)
+        mpPlayer->setShift(s);
+    qDebug("===! setShift: %f",s);
+    emit shiftChanged();
+}
+
 AVPlayer* QmlAVPlayer::player()
 {
     return mpPlayer;
@@ -650,6 +666,27 @@ void QmlAVPlayer::_q_stopped()
     emit playbackStateChanged();
 }

+int QmlAVPlayer::aoBufferCount() const
+{
+    AudioOutput *ao = mpPlayer->audio();
+    if (!ao)
+        return -1;
+    return ao->bufferCount();
+}
+
+void QmlAVPlayer::setAOBufferCount(int s)
+{
+    AudioOutput *ao = mpPlayer->audio();
+    if (!ao)
+        return;
+
+    if (s<4)  s=4;
+    if (s>32) s=32;
+    qDebug("=== AO::setBufferCount: %i",s);
+    ao->setBufferCount(s);
+    emit aoBufferCountChanged();
+}
+
 void QmlAVPlayer::applyVolume()
 {
     AudioOutput *ao = mpPlayer->audio();
diff --git a/qml/Video.qml b/qml/Video.qml
index ff72d93..b152024 100644
--- a/qml/Video.qml
+++ b/qml/Video.qml
@@ -240,6 +240,20 @@ Item {
     property alias playbackRate:    player.playbackRate

     /*!
+        \qmlproperty real Video::shift
+
+        Shift AV
+    */
+    property alias shift:           player.shift
+
+    /*!
+        \qmlproperty real Video::aoBufferCount
+
+        AudioOutput BufferCount
+    */
+    property alias aoBufferCount:    player.aoBufferCount
+
+    /*!
         \qmlproperty int Video::position

         This property holds the current playback position in milliseconds.
diff --git a/qml/plugins.qmltypes b/qml/plugins.qmltypes
index c611284..f4550ff 100644
--- a/qml/plugins.qmltypes
+++ b/qml/plugins.qmltypes
@@ -188,6 +188,8 @@ Module {
         Property { name: "autoPlay"; type: "bool" }
         Property { name: "autoLoad"; type: "bool" }
         Property { name: "playbackRate"; type: "double" }
+        Property { name: "shift"; type: "double" }
+        Property { name: "aoBufferCount"; type: "int" }
         Property { name: "source"; type: "QUrl" }
         Property { name: "loops"; type: "int" }
         Property { name: "bufferProgress"; type: "double"; isReadonly: true }
diff --git a/src/AVPlayer.cpp b/src/AVPlayer.cpp
index 6674a7c..3e3146e 100644
--- a/src/AVPlayer.cpp
+++ b/src/AVPlayer.cpp
@@ -213,6 +213,22 @@ qreal AVPlayer::speed() const
     return d->speed;
 }

+void AVPlayer::setShift(qreal shift)
+{
+    if (qAbs(shift)<0.001) shift=0;
+    if (shift == d->av_shift)
+        return;
+    d->av_shift = shift;
+    if (d->vthread)
+        d->vthread->setAVShift(d->av_shift);
+    emit shiftChanged(d->av_shift);
+}
+
+qreal AVPlayer::shift() const
+{
+    return d->av_shift;
+}
+
 void AVPlayer::setInterruptTimeout(qint64 ms)
 {
     if (ms < 0LL)
@@ -1133,6 +1149,10 @@ void AVPlayer::playInternal()
         return;
         //return false;
     }
+    if (d->vthread)
+    {
+        d->vthread->setAVShift(d->av_shift);
+    }
     // setup clock before avthread.start() becuase avthreads use clock. after avthreads setup because of ao check
     if (d->last_position > 0) {//start_last) {
         masterClock()->pause(false); //external clock
diff --git a/src/AVPlayerPrivate.cpp b/src/AVPlayerPrivate.cpp
index 83a2a0b..b083a70 100644
--- a/src/AVPlayerPrivate.cpp
+++ b/src/AVPlayerPrivate.cpp
@@ -91,6 +91,7 @@ AVPlayer::Private::Private()
     , vthread(0)
     , vcapture(0)
     , speed(1.0)
+    , av_shift(0)
     , ao_enabled(true)
     , vos(0)
     , aos(0)
diff --git a/src/AVPlayerPrivate.h b/src/AVPlayerPrivate.h
index 1de46b4..2e7e8ac 100644
--- a/src/AVPlayerPrivate.h
+++ b/src/AVPlayerPrivate.h
@@ -138,6 +138,7 @@ public:
     VideoCapture *vcapture;
     Statistics statistics;
     qreal speed;
+    qreal av_shift;
     bool ao_enabled;
     OutputSet *vos, *aos;
     QVector<VideoDecoderId> vc_ids;
diff --git a/src/QtAV/AVPlayer.h b/src/QtAV/AVPlayer.h
index eb26baf..4450c7d 100644
--- a/src/QtAV/AVPlayer.h
+++ b/src/QtAV/AVPlayer.h
@@ -252,6 +252,12 @@ public:
     qreal speed() const;

     /*!
+     * shift video to audio
+     */
+    void setShift(qreal shift);
+    qreal shift() const;
+
+    /*!
      * \brief setInterruptTimeout
      * Emit error(usually network error) if open/read spends too much time.
      * If isInterruptOnTimeout() is true, abort current operation and stop playback
@@ -439,6 +445,7 @@ Q_SIGNALS:
     void started();
     void stopped();
     void speedChanged(qreal speed);
+    void shiftChanged(qreal shift);
     void repeatChanged(int r);
     void currentRepeatChanged(int r);
     void startPositionChanged(qint64 position);
diff --git a/src/VideoThread.cpp b/src/VideoThread.cpp
index 1a2e185..471789f 100644
--- a/src/VideoThread.cpp
+++ b/src/VideoThread.cpp
@@ -43,11 +43,13 @@ public:
         AVThreadPrivate()
       , force_fps(-1)
       , force_dt(-1)
+      , av_shift(0)
       , last_deliver_time(0)
       , capture(0)
       , filter_context(0)
     {
     }
+
     ~VideoThreadPrivate() {
         //not neccesary context is managed by filters.
         filter_context = 0;
@@ -58,6 +60,7 @@ public:
     // not const.
     int force_dt; //unit: ms. force_fps = 1/force_dt.  <=0: ignore
     qint64 last_deliver_time;
+    qreal av_shift;

     double pts; //current decoded pts. for capture. TODO: remove
     VideoCapture *capture;
@@ -179,6 +182,12 @@ void VideoThread::setEQ(int b, int c, int s)
     }
 }

+void VideoThread::setAVShift(qreal shift)
+{
+    DPTR_D(VideoThread);
+    d.av_shift = shift;
+}
+
 void VideoThread::applyFilters(VideoFrame &frame)
 {
     DPTR_D(VideoThread);
@@ -245,6 +254,8 @@ bool VideoThread::deliverVideoFrame(VideoFrame &frame)
     return true;
 }

+
+
 //TODO: if output is null or dummy, the use duration to wait
 void VideoThread::run()
 {
@@ -282,6 +293,8 @@ void VideoThread::run()
     bool sync_video = d.clock->clockType() == AVClock::VideoClock; // no frame drop
     const qint64 start_time = QDateTime::currentMSecsSinceEpoch();
     bool skip_render = false; // keep true if decoded frame does not reach desired time
+    qreal av_shift_current=0;
+
     while (!d.stop) {
         processNextTask();
         //TODO: why put it at the end of loop then playNextFrame() not work?
@@ -320,9 +333,22 @@ void VideoThread::run()
             sync_audio = false;
             sync_video = false;
         }
-        const qreal dts = pkt.dts; //FIXME: pts and dts
+
+        ///////////////////////////////////////////////////////////
+        qreal av_shift_diff = d.av_shift - av_shift_current;
+        if (qAbs(av_shift_diff)>0.01)
+        {
+            av_shift_current+=(av_shift_diff>0?0.01:-0.01);
+            qDebug("=== shifting ===");
+        }
+        ///////////////////////////////////////////////////////////
+
+        const qreal dts = pkt.dts + av_shift_current; //FIXME: pts and dts
         // TODO: delta ref time
         qreal diff = dts - d.clock->value();
+
+        qDebug("=== av_shift %f (%f), diff:%f, d.delay:%f",av_shift_current,d.av_shift,diff,d.delay);
+
         if (diff < 0 && sync_video)
             diff = 0; // this ensures no frame drop
         if (diff > kSyncThreshold) {
@@ -405,6 +431,7 @@ void VideoThread::run()
                 diff = 0;
             }
         }
+
         //audio packet not cleaned up?
         if (diff > 0 && diff < 1.0 && !seeking) {
             // can not change d.delay here! we need it to comapre to next loop
@@ -465,6 +492,7 @@ void VideoThread::run()
         }
         if (frame.timestamp() == 0)
             frame.setTimestamp(pkt.pts); // pkt.pts is wrong. >= real timestamp
+
         const qreal pts = frame.timestamp();
         // seek finished because we can ensure no packet before seek decoded when render_pts0 is set
         //qDebug("pts0: %f, pts: %f", d.render_pts0, pts);
diff --git a/src/VideoThread.h b/src/VideoThread.h
index ad2f99c..5f5682c 100644
--- a/src/VideoThread.h
+++ b/src/VideoThread.h
@@ -46,6 +46,7 @@ public:
     void setContrast(int val);
     void setSaturation(int val);
     void setEQ(int b, int c, int s);
+    void setAVShift(qreal shift);

 public Q_SLOTS:
     void addCaptureTask();

[wang-bin]

can you try the latest code? I fixed a similar issue in bbce810

[Ligverd]

I tried the code commit 772c2a8c5 With standard playback buffer HTTP and twitches. Increases setBufferSize or setBufferCount sound jerky stops, but it turns 200ms lag

Ie the audio is shifted to the buffer size when changing the audio buffer size than the BufferCount 8 or BufferSize 1024 * 4

Issue is relevant :(

[Ligverd]

For testing sync AV http://77.241.16.66/pub/avsync.mp4

[sk2212]

I also noticed these issues @Ligverd descriped for Android 4.2.2 devices. Playing Upnp live stream content from a set up box over network audio and video were not playing syncron.

Did not try it with your updated version from commit @bbce810 @wang-bin and without increasing buffer size.

I will give @Ligverd patch a try.

[Ligverd]

I tried bbce810 with my patch and without, with different buffers.

Found such a thing! When you increase setBufferSize or setBufferCount in AudioOutput sound gets pretty late and the larger the buffer the more the more the lag. BUT, if you increase both buffers then all will be fine and the sound stutters, and no resynchronization. It is only necessary to find the optimal buffer sizes and try different modes, and also to pull in QML the management of these buffers.

Working code to play a network stream.

player->audio()->setBufferSize(1024*4);
player->audio()->setBufferCount(16);
player->setBufferMode(BufferPackets);
player->setBufferValue(16);

Resynchronization is very small, and perhaps it is necessary to choose the parameters more precisely.

[sk2212]

Uhh...did you know how to set the values in qml?

[Ligverd]

Now this can not be done, but make the patch I can.

[sk2212]

So should I apply your patch and than set the values setBufferSize, setBufferCount etc. from your comment above in QML?

In which file must I integrate

player->audio()->setBufferSize(1024*4); player->audio()->setBufferCount(16); player->setBufferMode(BufferPackets); player->setBufferValue(16);

which should work without your patch?!

[Ligverd]

These lines are not intended to QML there is no access to buffer management. To manage buffers need this functionality to add to the QML part.

For the test can take sampleplayer from the examples before mplayer->play() and add this code. The patch I will do later, I'm at a party and I am writing not from my computer.

[wang-bin]

Hi all, check out the latest code. I just fixed an opensl issue #231 . Now opensl is the default audio renderer and openal is no longer required.

: )

[sk2212]

Amazing...will check it tomorrow :-)!

The update should not affect any ffmpeg build, right? I just have to rebuild the QT plugin QtAv, haven`t I?

[Ligverd]

This is very good news!!!

Tried with default settings, everything works perfectly! Alleluia!

[Ligverd]

@sk2212 yes. Only last pull and rebuild QtAV