[BUG] Huawei's IVS video platform, using RTP to push streams, only receives audio data, but does not parse video data.

sling2007 commented 1 week ago

Phenomenon Description

Calling Huawei IVS video platform (uniformly using rtsp transmission in h264 format), zlmediakit's stream pulling and playback are normal. However, when IVS pushes streams to zlmediakit's rtp 10000 port, zlmediakit's console only sees audio data, but no video data. Only video is needed in the business.

IVS's request and response are as follows. zlmediakit's page and 10000 port packet capture are also as follows. zlmediakit configuration is at the end.

How to reproduce?

Expect to parse out video data and play successfully.

Related logs or screenshots

Postman request to IVS return:

zlmediakit monitoring page: only audio, no video

Wireshark port monitoring:

Expand to view detailed logs


```
#Please paste the detailed log here!
```

Configuration

RTP and RTSP related configurations are as follows. [rtmp] directProxy=1 enhanced=0 handshakeSecond=15 keepAliveSecond=15 port=1935 sslport=0

[rtp] audioMtuSize=600 h264_stap_a=1 lowLatency=0 rtpMaxSize=10 videoMtuSize=1500

[rtp_proxy] dumpDir= gop_cache=1 h264_pt=99 h265_pt=99 opus_pt=100 port=10000 port_range=30000-35000 ps_pt=99 rtp_g711_dur_ms=100 timeoutSec=15 udp_recv_socket_buffer=4194304

[rtsp] authBasic=0 directProxy=0 handshakeSecond=15 keepAliveSecond=15 lowLatency=0 port=554 rtpTransportType=-1 sslport=0

[shell] maxReqSize=1024 port=0

Please expand to view detailed configuration


```ini
#Please paste the contents of the config.ini file here!
```

Various environment information

Code commit record/git commit hash:
Operating system and version:
Hardware information:

crash backtrace:

#Paste the crash information backtrace here

Other information to be supplemented:

现象描述

调用华为IVS视频平台(统一使用了rtsp传输的h264格式)，zlmediakit的拉流、播放正常。但从IVS推流到zlmediakit的rtp 10000端口时，zlmediakit控制台只看到了audio数据，没有video数据。业务中只需要video即可。

IVS的请求和响应，如下图所示。 zlmediakit的页面和10000端口抓包，也如下图。 zlmediakit配置见最后。

如何复现?

期望能解析出video数据，并播放成功。

配置

RTP和RTSP的相关配置，如下图。 [rtmp] directProxy=1 enhanced=0 handshakeSecond=15 keepAliveSecond=15 port=1935 sslport=0

[rtp] audioMtuSize=600 h264_stap_a=1 lowLatency=0 rtpMaxSize=10 videoMtuSize=1500

[rtp_proxy] dumpDir= gop_cache=1 h264_pt=99 h265_pt=99 opus_pt=100 port=10000 port_range=30000-35000 ps_pt=99 rtp_g711_dur_ms=100 timeoutSec=15 udp_recv_socket_buffer=4194304

[rtsp] authBasic=0 directProxy=0 handshakeSecond=15 keepAliveSecond=15 lowLatency=0 port=554 rtpTransportType=-1 sslport=0

[shell] maxReqSize=1024 port=0

展开查看详细配置


```ini
#config.ini内容粘在这里!
```

各种环境信息

代码提交记录/git commit hash:
操作系统及版本:
硬件信息:
crash backtrace:
```
#崩溃信息backtrace粘贴至此
```
其他需要补充的信息:

TRANS_BY_GITHUB_AI_ASSISTANT

sling2007 commented 1 week ago

In zlmediakit, the ports for receiving audio and video are both 10000, is that correct?
I tried changing the audio to 30010, but this way the video still can't be parsed.

zlmediakit中，接收audio和video的端口都是10000，这个对吗？
我试过把audio改成30010，但这样也解析不到视频。

TRANS_BY_GITHUB_AI_ASSISTANT

xia-chu commented 1 week ago

Please provide the log and version number of the following zlm

请提供下zlm的日志和版本号

TRANS_BY_GITHUB_AI_ASSISTANT

sling2007 commented 1 week ago

zlm's version number: Branch: master Build time: 2024-04-21T06:43:54 Version: 664d0b6 zml's log is as follows (the rest of the repeated screens are not posted)

2024-10-24 13:44:12.252 T [MediaServer] [1-event poller 33] UdpServer.h:40 operator() | 8559-588(10.101.25.204:17784) mediakit::RtpSession
2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 23] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread
2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 23] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread
    Last message repeated 9 times
2024-10-24 13:44:12.252 I [MediaServer] [1-event poller 33] RtpProcess.cpp:269 operator() | 00000000(10.101.25.204:17784) Allow RTP streaming
2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 33] H265Rtp.cpp:215 decodeRtp | Does not support this type of 265 RTP packet, nal type 51, rtp:
version:2
padding:0
ext:20
csrc:0
mark:0
pt:99
seq:1
stamp:3386346273
ssrc:0
rtp size:52
payload offset:24
payload size:16

2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 33] H265Rtp.cpp:215 decodeRtp | Does not support this type of 265 RTP packet, nal type 52, rtp:
version:2
padding:0
ext:20
csrc:0
mark:0
pt:99
seq:2
stamp:3386346273
ssrc:0
rtp size:41
payload offset:24
payload size:5

2024-10-24 13:44:12.291 W [MediaServer] [1-event poller 33] H265Rtp.cpp:90 unpackAp | invalid rtp data size:613 < 16384,rtp:
version:2
padding:0
ext:20
csrc:0
mark:1
pt:99
seq:14
stamp:3386353479
ssrc:0
rtp size:653
payload offset:24
payload size:617

2024-10-24 13:44:12.311 W [MediaServer] [1-event poller 33] H265Rtp.cpp:90 unpackAp | invalid rtp data size:651 < 24576,rtp:
version:2
padding:0
ext:20
csrc:0
mark:1
pt:99
seq:15
stamp:3386357073
ssrc:0
rtp size:691
payload offset:24
payload size:655

zlm的版本号：分支：master 构建时间: 2024-04-21T06:43:54 版本: 664d0b6 zml的日志如下（其余重复刷屏的没有贴出来）


2024-10-24 13:44:12.252 T [MediaServer] [1-event poller 33] UdpServer.h:40 operator() | 8559-588(10.101.25.204:17784) mediakit::RtpSession
2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 23] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread
2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 23] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread
Last message repeated 9 times
2024-10-24 13:44:12.252 I [MediaServer] [1-event poller 33] RtpProcess.cpp:269 operator() | 00000000(10.101.25.204:17784) 允许RTP推流
2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 33] H265Rtp.cpp:215 decodeRtp | 不支持该类型的265 RTP包, nal type51, rtp:
version:2
padding:0
ext:20
csrc:0
mark:0
pt:99
seq:1
stamp:3386346273
ssrc:0
rtp size:52
payload offset:24
payload size:16

2024-10-24 13:44:12.252 W [MediaServer] [1-event poller 33] H265Rtp.cpp:215 decodeRtp | 不支持该类型的265 RTP包, nal type52, rtp: version:2 padding:0 ext:20 csrc:0 mark:0 pt:99 seq:2 stamp:3386346273 ssrc:0 rtp size:41 payload offset:24 payload size:5

2024-10-24 13:44:12.291 W [MediaServer] [1-event poller 33] H265Rtp.cpp:90 unpackAp | invalid rtp data size:613 < 16384,rtp: version:2 padding:0 ext:20 csrc:0 mark:1 pt:99 seq:14 stamp:3386353479 ssrc:0 rtp size:653 payload offset:24 payload size:617

2024-10-24 13:44:12.311 W [MediaServer] [1-event poller 33] H265Rtp.cpp:90 unpackAp | invalid rtp data size:651 < 24576,rtp: version:2 padding:0 ext:20 csrc:0 mark:1 pt:99 seq:15 stamp:3386357073 ssrc:0 rtp size:691 payload offset:24 payload size:655



`TRANS_BY_GITHUB_AI_ASSISTANT`

sling2007 commented 1 week ago

This log is written as 265, however I have set the camera, IVS, etc. to 264 along the way, so it should be using 264 encoding,
and also, on the zml page, when the stream is playing normally, I see that it is 264.

这个日志写的265，不过我在摄像头、IVS等一路设置为264了，用的应该是264编码吧，
而且zml页面中，拉流正常播放时，看到是264了。

TRANS_BY_GITHUB_AI_ASSISTANT

sling2007 commented 1 week ago

Additionally, using VLC's sdp file, it is possible to play this stream in VLC. （Removing audio and only keeping video also works.）

m=video 21202 RTP/AVP 99
a=rtpmap:99 H264
m=audio 21203 RTP/AVP 0  
a=rtpmap:0 PCMU/8000
a=framerate:25
c=IN IP4 10.101.59.170

So, can you help me take a look, is it because I configured something wrong in zlm? Or do I need to upgrade the version? Or is there a problem with the data format of this stream?

另外用VLC的sdp文件，是可以在VLC里播放这个推流的。（去掉audio，只保留video也是可以播放的。）

m=video 21202 RTP/AVP 99
a=rtpmap:99 H264
m=audio 21203 RTP/AVP 0  
a=rtpmap:0 PCMU/8000
a=framerate:25
c=IN IP4 10.101.59.170

所以帮忙看一下，是我在zlm哪里配置错了？还是需要升级版本呢？还是这个推流的数据格式有问题呢？

TRANS_BY_GITHUB_AI_ASSISTANT

xia-chu commented 1 week ago

zlm's port, although it can receive ES streams, does not support receiving multiple ES streams on one port.

zlm的端口虽然可以接收es流但是不支持一个端口收多路es流。

TRANS_BY_GITHUB_AI_ASSISTANT

xia-chu commented 1 week ago

You can use two ports to receive audio and video streams separately, however, zlm will not merge them into one stream.

你可以用两个端口分别接收音视频流不过zlm不会把他们合并为一个流

TRANS_BY_GITHUB_AI_ASSISTANT

sling2007 commented 6 days ago

Thank you for your reply and suggestion. I just tested it, 1、Using zlm's 10000 port to receive video, 30001 to receive audio, the push stream configuration is as follows: 2、Currently, no streams can be seen on zlm's monitoring page. 3、However, UDP data packets received on port 10000 can still be seen in Wireshark. 4、The error reported in the zlm log is similar to yesterday's, which are "does not support this type of 265 RTP package" and "unpackAp | invalid rtp data size:651 < 24576,rtp:".

So is there another way for me to try, such as configuring the rtp size in zlm? For example, use ffmpeg to process it first and then convert it to zlm?

The log is as follows: 2024-10-25 09:28:51.199 T [MediaServer] [1-event poller 26] UdpServer.h:40 operator() | 11539-589(10.101.25.204:17800) mediakit::RtpSession 2024-10-25 09:28:51.199 W [MediaServer] [1-event poller 14] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread 2024-10-25 09:28:51.199 W [MediaServer] [1-event poller 14] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread^M Last message repeated 8 times 2024-10-25 09:28:51.200 I [MediaServer] [1-event poller 26] RtpProcess.cpp:269 operator() | 00000000(10.101.25.204:17800) Allow RTP push stream 2024-10-25 09:28:51.200 W [MediaServer] [1-event poller 26] H265Rtp.cpp:215 decodeRtp | Does not support this type of 265 RTP package, nal type51, rtp:^M version:2^M padding:0^M ext:20^M csrc:0^M mark:0^M pt:99^M seq:1^M stamp:3386346273^M ssrc:0^M rtp size:52^M payload offset:24^M payload size:16^M

2024-10-25 09:28:51.258 W [MediaServer] [1-event poller 26] H265Rtp.cpp:90 unpackAp | invalid rtp data size:651 < 24576,rtp:^M version:2^M padding:0^M ext:20^M csrc:0^M mark:1^M pt:99^M seq:15^M stamp:3386357073^M ssrc:0^M rtp size:691^M payload offset:24^M payload size:655^M

多谢回复和建议。刚测了一下， 1、使用zlm的10000端口接收video， 30001接收audio，推流配置如下： 2、目前zlm的监控页面上，看不到任何流了。 3、但是wireshark上还能看见，10000端口收到的UDP数据包。 4、zlm日志里报的错误和昨天类似，分别是“不支持该类型的265 RTP包”和“unpackAp | invalid rtp data size:651 < 24576,rtp:”。

所以有没有别的思路让我可以试试，比如配置zlm里的rtp size？比如用ffmpeg先处理，再转到zlm？

日志如下： 2024-10-25 09:28:51.199 T [MediaServer] [1-event poller 26] UdpServer.h:40 operator() | 11539-589(10.101.25.204:17800) mediakit::RtpSession 2024-10-25 09:28:51.199 W [MediaServer] [1-event poller 14] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread 2024-10-25 09:28:51.199 W [MediaServer] [1-event poller 14] UdpServer.cpp:170 onRead_l | UDP packet incoming from other thread^M Last message repeated 8 times 2024-10-25 09:28:51.200 I [MediaServer] [1-event poller 26] RtpProcess.cpp:269 operator() | 00000000(10.101.25.204:17800) 允许RTP推流 2024-10-25 09:28:51.200 W [MediaServer] [1-event poller 26] H265Rtp.cpp:215 decodeRtp | 不支持该类型的265 RTP包, nal type51, rtp:^M version:2^M padding:0^M ext:20^M csrc:0^M mark:0^M pt:99^M seq:1^M stamp:3386346273^M ssrc:0^M rtp size:52^M payload offset:24^M payload size:16^M

2024-10-25 09:28:51.258 W [MediaServer] [1-event poller 26] H265Rtp.cpp:90 unpackAp | invalid rtp data size:651 < 24576,rtp:^M version:2^M padding:0^M ext:20^M csrc:0^M mark:1^M pt:99^M seq:15^M stamp:3386357073^M ssrc:0^M rtp size:691^M payload offset:24^M payload size:655^M

TRANS_BY_GITHUB_AI_ASSISTANT

xia-chu commented 6 days ago

Are you sure Huawei's video es stream is H.265?

你确定华为的视频es流就是265的？

TRANS_BY_GITHUB_AI_ASSISTANT

xia-chu commented 6 days ago

Configure the profile as follows:

Change H264 to 99, and change H265 to something else, such as 102

配置文件这里改下：

H264改成99，H265改成其他，比如说102

TRANS_BY_GITHUB_AI_ASSISTANT

sling2007 commented 6 days ago

Thank you, after modifying the value of this Payload Type (H264 changed to 99, H265 changed to another, for example 102), it indeed worked. I confirm that I am using h264 encoding. Can't the PT values of h264 and h265 be the same?

多谢，修改了这个Payload Type的值（H264改成99，H265改成其他，比如说102）后，的确可以了。我确认是用的h264编码。 h264和h265的pt值，不可以一样吗？

TRANS_BY_GITHUB_AI_ASSISTANT

ZLMediaKit / ZLMediaKit