Android muxer h264 aac

[FunnyDevs]

Hello! Could i use xuggler to mux encoded data (h264 and aac encodec bytebuffers) from Android camera to file? Because i don't find an example about this.

Thank you

[olivierayache]

Yes you can do that by using ByteBuffer or even for better performance a Surface with getHardwareSurface() on IStreamCoder

Configure your Camera2 API by following this https://developer.android.com/reference/android/hardware/camera2/CameraDevice#createCaptureSession(android.hardware.camera2.params.SessionConfiguration)

getHardwareSurface() will be the equivalent of createInputSurface() from MediaCodec

[FunnyDevs]

I am using bytebuffers (i'm trying to edit a library that gets me encoded data), this is my code

public void encodeVideoByteBuffer(ByteBuffer buffer, MediaCodec.BufferInfo bufferInfo) {}

public void encodeAudioByteBuffer(ByteBuffer buffer, MediaCodec.BufferInfo bufferInfo) {}

So i have two methods where i have h264 data (encodeVideoByteBuffer) and aac data (encodeAudioByteBuffer). How can i do the muxing with Xuggler? I haven't found a snippet with real time data

[olivierayache]

So your datas are already encoded and you want to mux them that is it?

You should be able to write you ByteBuffers directly in IContainer object via wrapping your datas in IPacket and write them in IContainer via writePacket(IPacket p)

To create an IPacket for ByteBuffer you can do that

IBuffer buf = IBuffer.make(null, byteBuffer, 0, size) IPacket packet = IPacket.make(buf)

To write a packet in IContainer it must be complete, use setComplete() method on IPacket to do that

[FunnyDevs]

Yes, i must only mux them.

I encounter 2 problems now

1)

public void init() {
    try {

        file.createNewFile();
        iContainer = IContainer.make();
        IContainerFormat format = IContainerFormat.make();
        format.setInputFormat("mp4");
        if (iContainer.open(file.getAbsolutePath(), IContainer.Type.WRITE, format) <0)
            throw new RuntimeException("failed to open");

        ICodec videoCodec = ICodec.findEncodingCodec(ICodec.ID.AV_CODEC_ID_H264);
        IStream videoStream = iContainer.addNewStream(videoCodec);
        videoStreamCoder = videoStream.getStreamCoder();
        IRational frameRate = IRational.make(1, 30);

        videoStreamCoder.setWidth(1280);
        videoStreamCoder.setHeight(720);
        videoStreamCoder.setFrameRate(frameRate);
        videoStreamCoder.setTimeBase(IRational.make(frameRate.getDenominator(),
                frameRate.getNumerator()));
        videoStreamCoder.setPixelType(IPixelFormat.Type.YUV420P);

        if (iContainer.writeHeader() < 0) throw new RuntimeException();

    } catch (Throwable t) {
        t.printStackTrace();
    }
}

this is the init method of my muxer class. With this code i am unable to go on because, obviously, ICodec.findEncodingCodec(ICodec.ID.AV_CODEC_ID_H264) returns null, because your library is LGPL (and i don't want the ffmpeg GPL files).

So, how could i bypass this problem?

2) From the previous code, if i change AV_CODEC_ID_H264 to AV_CODEC_ID_MPEG4, i obtain this error Application provided invalid, non monotonically increasing dts to muxer in stream 0: 9223090561878065151 >= 64800000000

this is the complete "encodeVideoByteBuffer" code

 public void encodeVideoByteBuffer(ByteBuffer buffer, MediaCodec.BufferInfo bufferInfo) {

    IPacket packet = IPacket.make(IBuffer.make(
            null,buffer,0,bufferInfo.size
    ));
    packet.setTimeStamp( bufferInfo.presentationTimeUs );
    packet.setTimeBase( IRational.make(1,1000) );
    int pksz = packet.getSize();
    packet.setComplete(true,pksz);

    iContainer.writePacket(packet);

}

Where does it get ' 9223090561878065151 ' value?

[olivierayache]

Hello for the Android dependency you should use this one https://cloudsmith.io/~olivier-ayache/repos/xuggler/packages/detail/maven/xuggle-xuggler-android-all/5.7.0-20210702.222121-48/a=noarch;xg=xuggle/

It is LGPL and will allow you to use Android HW encoding

Maybe we could have a talk if you need some help to make it work

[FunnyDevs]

Yes, you would be doing me a big favour :). Let me know where we could have a talk (with the last build now ICodec.findEncodingCodec(ICodec.ID.AV_CODEC_ID_H264) returns a codec, but i have this errors when i try to write the metadata

[mov,mp4,m4a,3gp,3g2,mj2 @ 0xb400007bc5a2bf20] dimensions not set )

[olivierayache]

@FunnyDevs I just added you on LinkedIn we could continue by starting a talk as soon as you are available

[olivierayache]

One of my other project based on Xuggler https://github.com/olivierayache/xuggle-xuggler-android should fit your needs