VideoDecoder codec_input_provide sometimes runs out of buffer space

[IanCass]

02-14 12:54:18.857 29837-30239/ca.yyx.hu E/CAR.HU.J: [201] VideoDecoder.codec_input_provide | content.hasRemaining (): true capacity: 131072

There's code to handle this, but it doesn't work as expected

[IanCass]

Screen stops updating but is otherwise working. If you start tapping the screen randomly to try to get it to respond, you might do things you don't want to do like calling someone by mistake. Ask me how I know :D

[IanCass]

http://bigflake.com/mediacodec/

Q4. Can I stream data into the decoder? A4. Yes and no. The decoder takes a stream of "access units", which may not be a stream of bytes. For the video decoder, this means you need to preserve the "packet boundaries" established by the encoder (e.g. NAL units for H.264 video)

Perhaps this is the problem?

[IanCass]

Temporary fix in place. just drop the oversize video chunk. Causes temporary screen corruption but other than that continues to work.

[borconi]

I haven't looked through all the changes done between the fork I'm using and this Java version, but I'm using this for the video buffer:

private byte[] fixed_res_buf = new byte[65536 * 8];

Which is ugly-ish (512kb wasted memory) but I never run into problems with MediaCodec. Before passing it to the decoder I do:

ByteBuffer content = ByteBuffer.wrap(fixed_res_buf, 0, 65536 * 8);

It is in-efficient I know but it prevents any decoding glitches.

My buffer looks like this, so I never hit the capacity and never need to drop content (creating glitche): Content: java.nio.ByteArrayBuffer[position=0,limit=6016,capacity=524288]

[IanCass]

The issue is that the codec input buffers are too small sometimes. These are given to us by Android by calling mCodec.getInputBuffers(). I can't really resize these. On rare occasions, streamed video input data is too large. This usually happens when Google Maps is doing an automatic overview of the route. I've since changed framerate from 60fps to 30fps and I've not seen any video corruption since. This seems like a workaround rather than a fix.

[borconi]

I never use 60fps so that might explain it why I never saw it. Possible that actually AA doesn't send the correct header of the video stream? AFAIK the first unit should contain the data, and Media Codec will decide the buffer based on resolution, fps, color depth, etc so do you think it's possible that AA reports something wrongly? I think 60 fps is not officially supported anyway.

[borconi]

I have fixed this, with threading the whole process. Basically now I run 3 threads instead of one

Thread 1)

• Send command to AA
• Poll data from AA (use 75 milliseconds time out instead of 150)
• If data is video reconstruct it and add it to video buffer
• If data is audio add it to audio buffer

Thread 2)

• Loop and process anything from audio buffer

Thread 3)

• Read and process anything from video thread, with a few small gotcha's The dequeueInputBuffer is set to -1 (never time out!) The dequeueOutputBuffer timeout is set to 16666 (60fps!) And the BIGGEST GOTCHA, you might be decoding the video TOO FAST so you need to make sure your 3rd thread sleeps if it's running too fast but of course this doesn't always happen, however if it does it will scramble the video, this is a very hardcore version of what I'm implementing now:

while (!m_stopping) {                                            // Loop until stopping...

          byte[] videodata = videostream.toByteArray();
          videostream.reset();

          int videopos = 0;
          int videosize = videodata.length;

          while (videopos < videosize) {
            int nextslicelength = ByteBuffer.wrap(videodata, videopos, 4).getInt();
            if (nextslicelength < 0) {
              Log.e("HU-VIDEO", "Next slice has negative value!");
              break;
            }
            byte[] videoslice = new byte[nextslicelength];
            videopos = videopos + 4;
            System.arraycopy(videodata, videopos, videoslice, 0, nextslicelength);

            videopos = videopos + nextslicelength;

            if (((System.currentTimeMillis() - lastframe) < 17) && (lastframe != 0)) {
              try {
                Log.d("HU-DECODE", "Lastframe:" + lastframe + " Now: " + System.currentTimeMillis() + "Sleeping: " + (17 - (System.currentTimeMillis() - lastframe)));
                Thread.sleep(17 - (System.currentTimeMillis() - lastframe));
                Log.d("HU-DECODER", "Decoding too fast sleeping...");
              } catch (InterruptedException e) {
                e.printStackTrace();
              }
            }
            if (m_player !=null)
            try {
              lastframe = m_player.media_decode(videoslice, nextslicelength);
            }
            catch (Exception e)
            {

            }

          }

        if ((videosize == 0) || (m_player==null))
          try {
            Thread.sleep(10);
          } catch (InterruptedException e) {
            e.printStackTrace();
          }

      }

With above I'm able to poll over Wifi, run AA in 720p with 60fps and works smooth as butter on my Nexus 7 (2012)

Hope it helps!

[IanCass]

That may work for you, but on my hardware, the problem is simple :- Android Auto on the phone sends more video data than the hardware decoder on my Android SBC can consume which means the video data needs to be split. You can't do this arbitrarily and need to split at the correct point - which is what I've not done yet.

I'm currently working on a completely different approach for my in-car pc. I'm no longer using Android but now using Ubuntu with https://github.com/IanCassTwo/headunit. This seems to work perfectly.

I may come back to this at some point later in the year when I get a new car and start on a new carpc project.