Openavb_harness bandwidth stress testing issue

[stujeffrey016]

Hi everyone, I've been doing some stress testing in OpenAVB on i.MX6Q (Listener). I set the listener's bandwidth to 100M, and use an AVB Talker to send Class B mjpeg streaming to Listener. After that, I use packet builder to send the broadcast streaming(best effort), make the whole bandwidth more than 100M(overload). I thought that the broadcast packets will be dropped, and the Class B streaming will still work fine, but in this case, Listener shows the error message as below:

(openavb_harness:3185): CRITICAL : gst_adapter_push: assertion 'GST_IS_BUFFER (buf)' failed (openavb_harness:3185): CRITICAL : gst_jpeg_dec_fill_input_buffer: assertion 'dec-current_frame_map.data != NULL' failed (openavb_harness:3185): CRITICAL : gst_jpeg_dec_fill_input_buffer: assertion 'dec-current_frame_map.data != NULL' failed

[1503486928:713515128 OPENAVB MJPEG INTERFACE] ERROR: Mapping is wrong. Fragment timestamp should be 1586401565 instead of 16501182936 [1503486928:713515128 OPENAVB MJPEG INTERFACE] ERROR: Mapping is wrong. Fragment timestamp should be 1586401565 instead of 16501182936 [1503486928:713515128 OPENAVB MJPEG INTERFACE] ERROR: Mapping is wrong. Fragment timestamp should be 1586401565 instead of 16501182936 [1503486928:713515128 OPENAVB MJPEG INTERFACE] ERROR: Mapping is wrong. Fragment timestamp should be 1586401565 instead of 16501182936

These error messages seems like Listener lost some Class B streaming packet make the timestamp goes wrong. It also shows some mapping error in Gstreamer. Shouldn't the AVB Listener protect Class A,B streaming? What cause the openavb_harness program stop in this scenario?

[pinealservo]

There are probably several things going on here, but I don't have enough information to say for sure. What was the talker system, and how was it connected to the listener? Which endpoint was generating the broadcast best-effort traffic? Were there any switches between them, and were they AVB-capable switches?

Regardless of those, you seem to have some misconceptions. First of all, while the i.MX6Q has hardware timestamping support, it does not have any hardware traffic shaping support. Therefore, all traffic shaping has to be provided by some software layer. Linux has some software traffic shaping support, but neither openavb_harness nor the SRP daemon itself are set up to configure it. This means that an endpoint using just those applications won't shape anything outside of the transmit shaping performed by the avtp_pipeline code that openavb_harness uses-- this shaping is done purely via software timing and is vulnerable to poor system tuning.

Second, you are talking about configuring the listener for a certain amount of bandwidth, but the standard AVB traffic shaping responsibility is on the talker and forwarding bridges, not the listener. Your Talker or some link between it and the Listener was almost certainly not configured as a proper AVB node if the streaming reservation traffic was not given priority over the best-effort traffic on the link to the Listener.

As far as the mjpeg traffic and gstreamer go, I'm afraid I haven't used any of the gstreamer interfaces enough to say exactly why it's having trouble, but it's entirely likely that your guess about missing packets is correct. The listener shouldn't have started to begin with if it wasn't connected to another AVB node, and if its peer claimed to be an AVB node and then sent more traffic than AVB would allow, that's not the listener's fault.

[stujeffrey016]

Hi @pinealservo, This is my testing architecture:

Talker has full AVB capability, and there is an AVB Switch between Talker and Listener. I thought that Listener don't need the traffic-shaping part, but in this case it seems need to provide some packet handling to guarantee the Class B streaming packet. In my opinion, I thought the AVB standard is build for guarantee the AVB streaming, if the bandwidth is being overload, the stream still can display in the Listener. So I do this test is to make sure my conception about the protocol. Or AVB standard is just for make sure the latency of streaming?

[pinealservo]

If your talker and switch are behaving correctly, the talker's stream reservation traffic will be given higher priority in forwarding to the listener than the broadcast best-effort traffic. If this is the case and your listener still has trouble, then maybe it's not capable of handling packets at the full link speed of the connection it has to the switch. You may be able to prioritize ingress traffic on the streaming VLAN; I don't remember the details of the hardware or kernel driver with respect to support for multiple ingress queues.

Before jumping too deep into that, though, you should verify that the behavior you see is actually due to lost packets and not something else. In general, a correctly-functioning AVB system does protect streaming reservation traffic against non-streaming traffic, and I've tested much larger scenarios and much heavier overloads than your illustrated scenario. However, I'm not familiar with the the exact components you're using or how faithfully they implement the standards, so you will have to figure out where things are going wrong.

[stujeffrey016]

Hi @pinealservo ,

In the same testing scenario, I used the VPU to speed up the display, and I found out that Listener can still display the mjpeg stream, but the picture is broken. By the way, we can confirm that there's some packet lost in stress test, cause I got some error message like below:

AVTP sequence mismatch: expected: 234; got: 237; lost 3 AVTP sequence mismatch: expected: 239; got: 242; lost 3 AVTP sequence mismatch: expected: 44; got: 47; lost 3 AVTP sequence mismatch: expected: 145; got: 146; lost 1

I use the same testing scenario in PC base with I210, and I got the same result. I use Switch with broadcom chip between the Talker and Listener. I'm not sure is the Switch or Listener problem, cause we sent this Switch to AVnu testing before.

Thanks, Jeffrey.

[pinealservo]

I don't have enough information to guess where the problem is happening. You need to use diagnostic tools to figure out whether stream reservations are actually set up, whether packets are being dropped at talker or listener, what rate things are actually arriving at the switch and listener, etc. You might also try directed unicast traffic to aim at just the talker or just the listener to see whether the same result occurs, but you will ultimately still need to gather more information beyond AVTP sequence mismatch errors to diagnose the problem.

[stujeffrey016]

Hi @pinealservo ,

Does the i.MX6SoloX got the shaper ability? I built the openavb with generic tool chain in the older version.

Best regards, Jeffrey.

[pinealservo]

There is hardware shaping capability in the i.MX 6SoloX, but the code here does not use it because there is not a standard kernel interface to it yet and no one wants to create or maintain another separate out-of-tree driver for the i.MX network interfaces. I am working (along with several other groups) on getting these interfaces in place in Linux, but it will take a while.

Meanwhile, the avtp_pipeline includes a shaper daemon that configures the Linux software-based traffic shaping subsystem in a manner that roughly approximates the credit-based shaper behavior for one stream at a time. This requires no hardware support, so it works with the existing i.MX network driver. When hardware support is in place via standard kernel interfaces, the shaper daemon will probably be updated to use those if available.

[stujeffrey016]

Hi @pinealservo, Thanks for your information. Is the “credit-based shaper” make sure that my Class B sending rate’s value is a constant(250us) in this case? Cause in my opinion, credit-based shaper works in the situation of more than one stream. I don’t know whether shaper will effect this testing case or not. I found out that the issue of packet drop is because of my switch class B buffer queue is too small. But that doesn’t make sense, cause the reservation has been written in switch table, it will provide enough space for class B queue and make sure it won’t drop the packets. Is there a possible that my talker didn’t send packets in right sending rate? By the way, I use generic tool chain to built up my imx6solox talker.

Best regards, Jeffery .