Forward H264 over RTSP - Githubissues

anilmaddala commented 2 years ago

I have a H264 video stream coming in from a server as a byte[].

How can I forward it over RTSP? Is there a sample code I could refer to?

pedroSG94 commented 2 years ago

You can use directly RtspClient class. Example: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/rtsp/RtspCamera1.java

You only need to focus on this: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/rtsp/RtspCamera1.java#L192

ByteBuffer -> You need convert byte[] to ByteBuffer
MediaCodec.BufferInfo -> You can create an instance of it and set timestamp (ts in microseconds), flag (value 1 normal frame, 5 keyframe) and size (bytebuffer size in bytes) parameters depend of your buffer.

https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/rtsp/RtspCamera1.java#L187 You need call it a time to indicate videoInfo before or after connect method to stream. Normally, you can extract sps and pps from keyframes. You can use this method for it: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/c3afe264acd1715276a8119650557797678e6b18/encoder/src/main/java/com/pedro/encoder/video/VideoEncoder.java#L356

anilmaddala commented 2 years ago

Thank you! How can I determine the flags to be set?

pedroSG94 commented 2 years ago

As I told you 1 for "normal frames (no IDR)" and 5 for keyframes (IDR). You have a table with nal unit values here: https://yumichan.net/video-processing/video-compression/introduction-to-h264-nal-unit/ You can check nal unit type like here: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtsp/src/main/java/com/pedro/rtsp/rtp/packets/H264Packet.kt#L55 Remember avoid modify buffer (that code modify the buffer because extract the header). You can study the way to extact the header and adapt that code. The easier way is duplicate BytteBuffer class and use that duplicated buffer for this operation.

anilmaddala commented 2 years ago

Got it. I am using rtsp-simple-server to run the RTSP server. The H264 server also provides information if the current video byte[] is an IDR. Does this implementation look correct?

public void onVideoFrame(@NonNull byte[] frame, int size, boolean isIdr) {
    ByteBuffer h264Buffer = ByteBuffer.wrap(frame);
    info = new MediaCodec.BufferInfo();
    info.size = size;
    info.offset = 0;
    info.presentationTimeUs = System.nanoTime() / 1000 - presentTimeUs;
    if(isIdr  && !rtspClient.isStreaming()) {
        presentTimeUs = System.nanoTime() / 1000;
        info.flags = MediaCodec.BUFFER_FLAG_KEY_FRAME;
        Pair<ByteBuffer, ByteBuffer> videoData = decodeSpsPpsFromBuffer(h264Buffer, size);
        ByteBuffer newSps = videoData.first;
        ByteBuffer newPps = videoData.second;
        rtspClient.setVideoInfo(newSps, newPps, null);
        rtspClient.connect("<rtsp://<rtsp-server-IP>:8554/mystream");
    }
    rtspClient.sendVideo(h264Buffer, info);
}

pedroSG94 commented 2 years ago

Close but not correct at all. Try this:

  public void onVideoFrame(@NonNull byte[] frame, int size, boolean isIdr) {
    ByteBuffer h264Buffer = ByteBuffer.wrap(frame);
    info = new MediaCodec.BufferInfo();
    info.size = size;
    info.offset = 0;
    info.presentationTimeUs = System.nanoTime() / 1000 - presentTimeUs;
    if(isIdr) {
      info.flags = MediaCodec.BUFFER_FLAG_KEY_FRAME;
      if (!rtspClient.isStreaming()) {
        presentTimeUs = System.nanoTime() / 1000;
        Pair<ByteBuffer, ByteBuffer> videoData = decodeSpsPpsFromBuffer(h264Buffer, size);
        ByteBuffer newSps = videoData.first;
        ByteBuffer newPps = videoData.second;
        rtspClient.setVideoInfo(newSps, newPps, null);
        rtspClient.connect("<rtsp://<rtsp-server-IP>:8554/mystream");
      }
    }
    if (rtspClient.isStreaming()) {
      rtspClient.sendVideo(h264Buffer, info); 
    }
  }

With this you avoid send video until you connect and keyframe flag is correctly send (previously it is only set the first time)

anilmaddala commented 2 years ago

Thank you!. I tried your corrected code and no crashes but the output on the receiving player seems to be corrupt. This is the ffplay log. `Initialized opengl renderer. [rtsp @ 0x7f2750000b80] SDP:aq= 0KB vq= 0KB sq= 0B f=0/0
v=0 o=- 0 0 IN IP4 127.0.0.1 s=Stream c=IN IP4 0.0.0.0 t=0 0 m=video 0 RTP/AVP 96 a=rtpmap:96 H264/90000 a=fmtp:96 packetization-mode=1; sprop-parameter-sets=ZYiECTxGKAAIC8cAMcGIjVaI7xHaImvJPJycmpOTk4jWoj9JyckQoiTUksREr+I9pOSZScnEavJycROtSckYpOTklUnJMojvJxGtSckIG4jBZASAwGPiI0XaIjgefNRE4p9ESBUaXiM39ERa9JIXiMQ+8korETvURM95OTUR3iPR2SYT4jMwMY3ESi94iEAcnxCfSRouIxXeI19EYUaXk4j2kkLxE+8Ri/pJGySiskheTUm4iJFdScnJxGrVE1PU9ddPXXXXXXXXXWq666Jpa5O+aIvJ1111yddddcvXXXXJxMOY1Me5ZjqVzODLWpt+piZepHMy6mnvXUjk76eaMNyzSS09PXT08vWmuuuu+uttZ2Ftknddddaqepai6Wunrrrrrrrrrrvrrp6665u9dddddLXXXXXXXXJ1ycsEjZGJid6nya8z3pkOpTE1uJ96nrrdcnXTy87JanjmEP/19RtO6ZVJzTarl61XWOoF11kZJPXXWr1XU9dRNRNdddddPXXXXXXXXXT10t9dd8nXXT11111yc0deuuubvNCj9PKZpXRO94lS83klmYjKotMTvUvbLNu9Tzd65pr1qImbeZ16nimMIue6/UbTtplrczn73ydSF65N11ydS1113zdqmJ3F1E1F111DCrNinqRUwvXXXXfXXXT1111311zRF76YmuuuuuuuuuuuaCJP0s7m94nbvNt9MzL1RNbmZ71zSvVarklrriYyxEzNuVoaeuniGLtNMm9PXUia1XJLXXXUtdS11ENriMSHPS9dRtRNdc0MCf9L1PTCddddT1131109ddddc0VeuumLvrrk665Ouuuunmhp+iWQVvIzy8zEMepZ5t6uebeq5pV6R5u8vUykeaK9ES2U9dPMz3rmmTqttckqqQvUtdddZOad6rvrm3bA,ZQDwiIQEDxGKADHPk0hEhe8REijqSQ2PkzcmpNMmKycm4iN3kniInUR+k5IhSSxEVqSJURO/5OIjNREavpOSMUR3iJ1qTk4yGEhLV00/F7/NP8X1T/i9b9NPxEWogkoiJ7ycmpJZIUNxGJP9EafREorvEYrqSQL0AjC4AaDgx8RKXtEYgfeIleoiV3kkUksm4iV7yRCkxcRqjsRFCfLNET0vw//jRUv9ftiTj617xEWK2iJhL3iIn6I9REgo7ybYjtJO5OTk4iQV1JO5OTFYj1EfpOI1tNFCTnVmwvLyy3uXm/S9ZOu9NMfFR0qen5llRkblSZExRnXJqfipa6f0x8ktEE11111xUPFwqdNPxfA+Zlk37Jc5CnwR0RHibScRL6+MQ6RNP8XIkKan/FaVdNP0z1H1CSmf0vcTMyXiZl9TOM0L080t6ldc2vV07rk6euuZhJC6omaTJ6iZt3me8T2qRM0t666mTU9dbqeupHUtdRCT2i98+L9LxOT+k77x1VmwnFSKCG6fzTwHEfXMr1S2rtHi0KeLT/OpLIuddRK3IuSAim3+fmGjVc6U0/xaEtCLVt/lQk+Hm6pZkXFU98nfcYqRbW1i4xItTNv4mQmF0JZjX/0/yiYkQLF6y9MW5oQNy6Zj/S9PTuuutTeqeuubpUXmz/9MIH6iaepZvVdaqR1PXUjZeupb031ydTquohVyc+L9PtNJlYPmzpZkyyqImNXozDTLFTpp+Yl5RE/NPmzpJDWXvmiV6pa63e66RlUrvVrIt836p66764iI9XXUsi2sehn6f/jZC7FT/jMYbjL6f0wrNCzfohhHUqI5dyMtc2b9XJyMrk6iGzdiI1enp5s7FV11ImbbvT1tqWaTV9d9S3313yRN9RAu5Gi5eTbfLzRAK+W6JxaKkTjfuiEJEDg1VSLN6myZ1zCHrafLyRNS113M0uTk62ltPm/Vu+TqWuTvuR3IkTqLrrtbXTX/rwXBB+Kan/FSFwzdP5GE1Ujm97i66666dVI5or1Mqm/V09c016YmJiNTd1S111u++uTqV13ITklqepHgAAAAAWUAbiIhA88RqP8nJl4Qiv//T/+9/DJJcfvlSIwQZ0dER/NEbvJEGwkqklk0ZJB5XqSWTcm5NybT775M1HyTDvnu8ZZPLR7vcmkSdSSqTkmJjzYTaPv0m5NSaknk1JybhCM/T//qf+qVgiEC/r/8/wiLH0ROKO8RHivKIm1ETC/SYuSQvERL3iN6iIoSf0RLSyaiMX6MjAwAKYMAY+tfiIt2iJB9BeInFfoiYQ94jN6iJ7RGKN4j1EdojvEdpNyYhwRPqSIcmI8kTJySCsdGBoYl//0yyByMTksqd9//wRFSd50iJ/p5RXVS3GBndhJXU9dyJF7lvl671fUiRd3u8S493nIvuZ3y6vvuWSJrrV7uIJilnn1UQ05S4Tl5vV6m1qeV+1X1P68n/PXPCISqmF665or0vTK665pKVLp4SYx1r9a/6ouaTOmnvNNqup3XXWmtN9aa76lif1dZe+IihQAY7TI4O39e7VVI091mgTvV6rDA8SVtaT67kVSJFldziXHybqW+9SMsnffXXXXJ131EK+91uom+u5HcivpESJ0ix6Ffp//Ca/XPWRMXJLUU2umepGxGryM836Z/RTHVRUXzT6n23rm7zeU3aTklkkbUrZOs/NLeuu+uXqWo4lFtKWpb7kUvJI0s5wmfHxE977y4+8tk3Ut9999999y3333333113uXqZXuXqW++pHcsvfXXavU55/M6wiFOjmJ/+X/N+rpiZeJfeuunhJih9MPf73/qu5a0xPqXml2qQnU99buQvfJ3LXUja6nqeo4lk771ffe0I71lx99Tu9p95MUtddddddTFwvLz97lk7775OufV5eplUW76iFXe66ib6imiT331MkTpH1P1/PhEJdN6li6Z5ZlMz3m/TfqdRTEV3+uaX0vM95nJ2mlskkY11uuSWTrrdy31GNqepaiC4+9GsfVdc+Mq94WVJtKeu9X1LXUik663e5+9yzS3pa66kUnXNq99dd88y9e0TPinvrvuRy8k9avrm/Sd9driudT+fCIW7/zz+EdUwjUQxpiaetU8zEK1dRLriHdpOu5E11trLyRNRCROp663cQ2p1fgAAAABZQAqiIhASPJiQAC+PU+vSTqTkxWIz5V8RLeTkiC48VvUkqiJRfpOTknk5JlJqTkimn33u+5b5OSWTcnJI5MuvUnJnwkooxEtpJlJqTcnJKoiX0kyk5OTk3JIpOFIoCfc49P/9eHz1zwjAE8RFinRxiJVeIxfojF/QpEL//16IxAAPHABjDES6iJfpJC8RveIzdoje0nEbvJn5MKtBGIfUm4jVpJXEbvJuSRskqiJXaaIXq0i8swhwk5+p61Uxe8uH/VK771N+vl770ny98vL3y9dddcnct99zOSY+ElLhdV3nxdxd9dTKtycksnXXfe6iEiLHo36f/wmf1zqcFRr+X+mEa6dTRHqeae1c3qtzSWrrqV3u+XnlVpt3rrd9d99Yo33ECjWbrV88qvUjT5ZZe+IiV63vvd99998nL11Ir5O+53fe773XP+uOPi3LpLvl6i75etV3y99dd8nfL2upzz+p8FQU6n/CUrf/p/MopjYqMKv7WvmZz/dXUtbmk1Wq6y8+3epa3NM9SRN94rUTXcjk7mdZscQ08mOJScqviP0nPI10mptLpdy8rLLtE6kaLy8urlvuWuTuW+++++933yPXfUIG0m60iSqXaXctSOpVXe770hP6TrqVTSerteudT/0CUJX6df89c8I3mdPqYmpZpX6Jk3qRsRPauuaR5Vub1Usne6nvklruKbXUhMfJLer1efHy9SDKvFbixLhO+5jYXuUmElrrV9davvvvvuIaL33ydc0jXVK6lqWf9LGNOUQ4SRI0t72lqpXLEJOcTwnJyd8vXXXUtdrHoJF36f/wn+ulVgqJLWvpjS9dT1zTK1y1xMz1P3nneqkTUSfrm731LJ1y9SOpmi6l1JqsuFllkd7lz8k6vvV6n/V13z69XNEr1Lay836utTyeriP18/61tEdITccfHlwktTyS31CDRJ5O+XruRIvWpO4mWRXq9S9ddd9q89f9T5ExDGomaR3l601LNEO9bZp7XuWeTrrrqeuuTwAAAABZQA5iIhA88RigAG/SSJPkncnJqSRySism0I7yak1JyaiJ7RG7xEvpOTk4iN2k5OTkhRScnJycnJycnJyTyckYJcJPJqTk1Ed5JmiSqSRpyOIlFG0k4rJMpMmEiRcnJEKTFyRMm5IhySuTkniP0kJB3x8Nnh3xESIe8RifaI3tJxGFAFBeTJgjtJLJybiPUksnJxEt4iRfSdcnffXcjvn16fvPKvX3z/q675OTueuuuuu++f9ffczvvuIcuXVKpJb5euupFfXeLknvljFL1O65Zb775YpJdxd8vXXUtr///IoSBHugT8onPlSR4rNFNNUtS1ub1XXXXUtdc0j9e75/08t5/1rXWq61fer5eI/XycncQ0XrqWpFL1LUiS6zYXkll75JneTFpKJvvruISXUqvrrdd8vXXWpOpb3fL1331ENKWpXayLfJ111I773JLXW6661WoiX1SOup6666iler71UtcnfXXXfJ1331xH6tV1311u+upb5O+45JdRN8vcwrLIkSJvcvL3u+Tn7ydak5Ou+Xl5e+9ovfL1yddzNLrTcTffXWp5/VydddddddRCrVS1ydS1M77lcnfXW6766kV9dy111uXqZV311PfXJ33FJF71JLUg6rkSWkSd311ExEv99cvfffcY0uee99d9dd9wgBr0hcEeqFgbqgkS6ll61UtSK9V1K765v2AAAAAFlABGCIhA48RigAHfSbiJ/ScnJIpOTk5OTcRFu8nJMpOTcnJycmpNSckguTkkFZIkVkkckSKyRSkkNyTqTcm5NxEr1ESek3JuIlepJHJyaiP0m5OSZSTi5NySuSLcmpOTk5IpSckik1Jycm5OTk5IgunmIRaInFbyRLk1JuTk1EbtJqTk5JYjvJMpOsVuW++++++eW98/eTrvvrrdxDklk5Opb6kV9ddd8nXfJy9dd8vXJ13111EKotSd8nJ1PXXXJ33y99d9dRDn7Sd9dcktdSK5HXU9ddddxDbnrV8nXNE3rrmmvXXXfe766lrrvl5eXkeXqRS9cR+rcnLEOu5ZJZOuu+Xmk9U9cnciqJV8vXfJ111zTekib775O++uuu4xSdck9d6rrrk766kVd7m/VI65p/66666666mU36omb9L3PNN6upnfLy9PfW75euTuVycvXfJyddavl66lvl6677675OuaJfr5Ouuu++u+4x133u+uu++uu5nJ1qpa55P6lmm/ueSe+uuaR3vk5pvVKr66ll665OXqZycvK8nLzS+vlinUTJzy+vl65OTk63XfL113ydddT3ydzKTrqJvk5oj1ddd99cne75+97vvvrk66iFJLfPJ69VK6mFMvXUS4jvP+vvmk9fXXL1O5eaX1ddd9RTk5eTl63csvXXXUsnL113y9brvqMU0S/V3ydddcnfXfXNE+rvrrrrrrvrmmd5Ouut33yd9d8R/1quvAAAAABZQAVQiIQEDxGKAAVjvkyYTkxLgid6iO0nEa/k3JisnJimSRyYrJPEatET6k5JHJySOTcmpNSSqSKNyTqTckgh5NySuSVScnJEC5NSSyTuSZyTuSZSTOTFZIsVk5NSRCknUnJycnJOoiS8m4iLdpOTUnJycnJyckY5OSdScnJI5JnJqTUksR2k1JqTk1JqacV9JxP6TtZeeIX8RH7S9dRjl6kUvXfU6qWTrvk7iHJyd9RB+p5OXdbl6nrvliFUtcktcncil66lk75euuub1J33LJ1yddS1111IpOuuuupnfWq6lUnXffJ11ydddd9dd8vXfUY2XqVVK5eu5ZOTk675ebvfXXLy9SOTrVd8sirruJrvuZS9dy13y9c3qb9XXXJLL131zS+pa675O+uopyd6rrvvrrrcnJ1uuu++upVJ13y9cnUQpeTrqWpa6ib5OTrk5O+uXrk7jHL1qu+pHfLLUtS13y9d9d8vXXfXXXffNJ6+nqeu+TvrrvvvrvmiPV11FJnif65OpZOXqZSdRNd9dRrl5O4QckU251Uiqeuupa675etycnfLyTKXk65Ou+uXruWuu5a75YQTUS5Za75eutV131zd767Zb7774j9XXXXXXXfffJ3zSavwAAAAAFlABhiIhAo8RigACAjHHxGv5JlJqSWTcRMtREWrSckQKyTuSIckorJyTC5OTFZItycnJycksnJEOSdycmpOSMUkqkkckorJKKySKSRSaklFZJFJEqTiNaiP0nJIpOSICwq4suE5JVJyRCk1JLES6k1JxG/ScnJuSVycnJyRCk5OTiJXaTiO8nEatJySOSRSTqTk5u99bk6nrvrrl65OuuXqMUvUtS1Iq3JLXU8nXXXXfL1113y9RzvvcnU8vUtd9dxjqQ2iI218nfL11qt1311K66663fL111FOuuuu+563Jyd8nXXXXXL111Kr66765OXuIUvJPXfW5Ou+Trvk77676kVd8vUivd7rdRN9dd9SK4h1jiqLl65OXrvrdd9dMty33yd999ddRSqZNS1uu5a6666666666nrrrk665e+Trvrrrrvk675euuuu+Tk665OXqMUve5eSJrrrl71J1GE7i5eTvl6763J309S1Or5Oohyd9dd99SOuuutVK66665OI/Vz/qlvrrvl67665euXvvvrvrqRSdd9ddbvrl6775eohS8m65J661XfepYxV3FydT3y9d9dd9cj131111111HE66lvrueuupa6665OuuupHXXfUyqNvrk676665euXk75Ou++upnL3IqlVdcnL1Mq1fXU9d9Syd8sIKu4uTqe+Xrvrrvp7675OTk66jjc/e+uuSLqWuuvA= a=control:trackID=0 m=audio 0 RTP/AVP 97 a=rtpmap:97 MPEG4-GENERIC/32000/2 a=fmtp:97 profile-level-id=1; mode=AAC-hbr; config=1290; sizelength=13; indexlength=3; indexdeltalength=3 a=control:trackID=1

[rtsp @ 0x7f2750000b80] setting jitter buffer size to 0 0B f=0/0
[rtsp @ 0x7f2750000b80] setting jitter buffer size to 0 0B f=0/0
[h264 @ 0x7f2750007ae0] Invalid NAL unit 0, skipping. 0B f=0/0
Last message repeated 2 times [h264 @ 0x7f2750007ae0] Ignoring NAL type 5 in extradata Last message repeated 1 times [h264 @ 0x7f2750007ae0] Ignoring NAL type 18 in extradata [h264 @ 0x7f2750007ae0] Ignoring NAL type 5 in extradata [h264 @ 0x7f2750007ae0] Ignoring NAL type 29 in extradata [h264 @ 0x7f2750007ae0] Invalid NAL unit 0, skipping. 0B f=0/0
Last message repeated 2 times [h264 @ 0x7f2750007ae0] Ignoring NAL type 5 in extradata Last message repeated 1 times [h264 @ 0x7f2750007ae0] Ignoring NAL type 18 in extradata [h264 @ 0x7f2750007ae0] Ignoring NAL type 5 in extradata [h264 @ 0x7f2750007ae0] Ignoring NAL type 29 in extradata [h264 @ 0x7f2750007ae0] non-existing PPS 0 referenced 0B f=0/0
Last message repeated 1 times [h264 @ 0x7f2750007ae0] decode_slice_header error [h264 @ 0x7f2750007ae0] non-existing PPS 0 referenced [h264 @ 0x7f2750007ae0] decode_slice_header error [h264 @ 0x7f2750007ae0] non-existing PPS 0 referenced [h264 @ 0x7f2750007ae0] decode_slice_header error [h264 @ 0x7f2750007ae0] non-existing PPS 0 referenced [h264 @ 0x7f2750007ae0] decode_slice_header error [h264 @ 0x7f2750007ae0] non-existing PPS 0 referenced [h264 @ 0x7f2750007ae0] decode_slice_header error [h264 @ 0x7f2750007ae0] non-existing PPS 0 referenced [h264 @ 0x7f2750007ae0] decode_slice_header error` Any ideas? Also are there any tools that could help debugging this?

pedroSG94 commented 2 years ago

sprop-parameter-sets is too large. It seems incorrect so your SPS and PPS is not correct.

Try create a fake sps and pps using byte[] generated by your mobile using the same resolution: Start a stream using my app example (with the same resolution that your stream, you maybe need use prepareVideo method with parameters) and get sps and pps byte[] from the callback, copy it and replace the original for this one.

Anyway, the correct way should be extract it from bytebuffers of the incomming stream. You can check if you receive nal unit type 7 (sps) and 8 (pps) to get the correct values.

anilmaddala commented 2 years ago

Your right. Fixed my sps, pps buffers and it works now.

Thank you again for the excellent work and support!

pedroSG94 / RootEncoder

Forward H264 over RTSP #1033