Multiple streams with AVB_FEATURE_IGB

[Raman-Jamloki]

I have two Linux systems(Ubuntu 16.0) with I210 installed connected one on one. I am trying harness with multiple streams (2, 4, 7) and using h264_gst_talker/listener configuration.

Multiple streams works in below configurations. 1 avtp_pipeline.mk AVB_FEATURE_ENDPOINT ?= 1 IGB_LAUNCHTIME_ENABLED ?= 0 PLATFORM_TOOLCHAIN ?= generic

generic.cmake set ( AVB_FEATURE_IGB 0 ) set ( IGB_LAUNCHTIME_ENABLED 0 ) set ( AVB_FEATURE_PCAP 0 )

driver = igb ( igb_avb not used)

2 avtp_pipeline.mk AVB_FEATURE_ENDPOINT ?= 1 IGB_LAUNCHTIME_ENABLED ?= 0 PLATFORM_TOOLCHAIN ?= generic

generic.cmake set ( AVB_FEATURE_IGB 0 ) set ( IGB_LAUNCHTIME_ENABLED 0 ) set ( AVB_FEATURE_PCAP 1 )

driver igb ( igb_avb not used)

Here I was facing multiple crashes(double free, stack smashing and all). This required pcap_compile call protection in pcapRawsockRxMulticast function. Worked smoothly after that.

Below configuration is not working for me. Its either crashing (segmentation fault) or I cant see any video output.

avtp_pipeline.mk AVB_FEATURE_ENDPOINT ?= 1 IGB_LAUNCHTIME_ENABLED ?= 1 PLATFORM_TOOLCHAIN ?= x86_i210_linux

x86_i210_linux.cmake set ( AVB_FEATURE_IGB 1) set ( AVB_FEATURE_PCAP 1 )

driver (igb_avb)

command on listener side (for 2 streams) sudo ./openavb_harness -I -s 2 -d 0 -a a0:36:9f:2d:01:ad h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0 command on talker side (for 2 streams) sudo ./openavb_harness -I -s 2 -d 0 -a a0:36:9f:2d:01:ad h264_gst_talker.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0

Has anyone tried multiple streams with igb based raw sockets?

[Raman-Jamloki]

Its only listener side which crashes. Console output is below

sudo ./openavb_harness -I enp8s0 -s 2 -d 0 -a a0:36:9f:2d:01:ad h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0 endpointServerThread attaching to 0000:08:00.0 Opening: h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0,dest_addr=91:e0:f0:00:fe:00,stream_addr=a0:36:9f:2d:01:ad,stream_uid=0,ifname=enp8s0 Opening: h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0,dest_addr=91:e0:f0:00:fe:01,stream_addr=a0:36:9f:2d:01:ad,stream_uid=1,ifname=enp8s0 Configuring: h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0,dest_addr=91:e0:f0:00:fe:00,stream_addr=a0:36:9f:2d:01:ad,stream_uid=0,ifname=enp8s0 Configuring: h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0,dest_addr=91:e0:f0:00:fe:01,stream_addr=a0:36:9f:2d:01:ad,stream_uid=1,ifname=enp8s0 Starting: h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0,dest_addr=91:e0:f0:00:fe:00,stream_addr=a0:36:9f:2d:01:ad,stream_uid=0,ifname=enp8s0 Starting: h264_gst_listener.ini,sr_class=A,map_nv_tx_rate=8000,max_transit_usec=2000,report_seconds=0,dest_addr=91:e0:f0:00:fe:01,stream_addr=a0:36:9f:2d:01:ad,stream_uid=1,ifname=enp8s0 [1535392960:460993675 OPENAVB osalTime] INFO: local_time = 1535392960330846494 [1535392960:461004624 OPENAVB osalTime] INFO: ml_phoffset = 0, ls_phoffset = 129753816 [1535392960:461018192 OPENAVB osalTime] INFO: ml_freqffset = 1.000000, ls_freqoffset = 0.999974 [1535392960:461022406 OPENAVB Endpoint] INFO: AVTP Pipeline: 0.1.3 (Development) [1535392960:461229089 OPENAVB Talker / Listener] INFO: AVTP Pipeline: 0.1.3 (Development) [1535392960:469541824 OPENAVB HAL Ethernet] INFO: TX buffers: 40 [1535392960:469557313 OPENAVB HAL Ethernet] INFO: IGB launch time feature is ENABLED [1535392960:477864702 OPENAVB Talker / Listener] INFO: Looking up symbol for function: openavbMapH264Initialize [1535392960:477874271 OPENAVB Talker / Listener] INFO: Looking up symbol for function: openavbIntfH264RtpGstInitialize [1535392960:478153693 OPENAVB Talker / Listener] INFO: Looking up symbol for function: openavbMapH264Initialize [1535392960:478156569 OPENAVB Talker / Listener] INFO: Looking up symbol for function: openavbIntfH264RtpGstInitialize [1535392960:489737751 OPENAVB Endpoint SRP] INFO: detected domain Class A PRIO=3 VID=0002... [1535392960:489741918 OPENAVB Endpoint SRP] INFO: detected domain Class B PRIO=2 VID=0002... [1535392960:489743872 OPENAVB Endpoint SRP] INFO: detected domain Class A PRIO=3 VID=0002... [1535392960:489745415 OPENAVB Endpoint SRP] INFO: detected domain Class B PRIO=2 VID=0002... [1535392960:490380097 OPENAVB Listener] INFO: Attach a0:36:9f:2d:01:ad/0 [1535392960:490535731 OPENAVB Listener] INFO: Attach a0:36:9f:2d:01:ad/1 [1535392961:243437714 OPENAVB Endpoint] INFO: SRP listener callback uid=0: tlkrDecl=1 [1535392961:243464092 OPENAVB Endpoint] INFO: SRP listener callback uid=1: tlkrDecl=1 [1535392961:243592799 OPENAVB Listener] INFO: Starting stream: a0:36:9f:2d:01:ad/1 [1535392961:243610213 OPENAVB Listener] INFO: Starting stream: a0:36:9f:2d:01:ad/0 error: XDG_RUNTIME_DIR not set in the environment. [1535392961:282417014 OPENAVB H264 Interface] INFO: Pipeline: appsrc name=avbsrc ! application/x-rtp,media=video,clock-rate=90000,encoding-name=H264,payload=96,ssrc=5,clock-base=1,seqnum-base=1 ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink sync=f[1535392961:282553099 OPENAVB H264 Interface] INFO: Pipeline: appsrc name=avbsrc ! application/x-rtp,media=video,clock-rate=90000,encoding-name=H264,payload=96,ssrc=5,clock-base=1,seqnum-base=1 ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink sync=f[1535392961:343862059 OPENAVB Endpoint] INFO: SRP listener callback uid=0: tlkrDecl=1 [1535392961:343883990 OPENAVB Endpoint] INFO: SRP listener callback uid=1: tlkrDecl=1 libva info: VA-API version 0.39.0 libva info: va_getDriverName() returns 0 libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nouveau_drv_video.so libva info: Found init function __vaDriverInit_0_39 libva info: va_openDriver() returns 0 error: XDG_RUNTIME_DIR not set in the environment. [1535392961:498318710 OPENAVB Raw Socket] INFO: Using igb implementation Segmentation fault (core dumped)

[Raman-Jamloki]

It works fine with single stream.

[Raman-Jamloki]

GDB trace

Thread 7 "openavb_harness" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7f84700 (LWP 6818)] pcapRawsockGetRxFrame (pvRawsock=0x7fffe00481f0, timeout=, offset=0x7ffff7f83de4, len=0x7ffff7f83de8) at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/platform/Linux/rawsock/pcap_rawsock.c:202 202 *len = rawsock->rxHeader->caplen; (gdb) backtrace

0 pcapRawsockGetRxFrame (pvRawsock=0x7fffe00481f0, timeout=, offset=0x7ffff7f83de4, len=0x7ffff7f83de8)

at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/platform/Linux/rawsock/pcap_rawsock.c:202

1 0x00000000004283fc in avtpTryRx (pStream=)

at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/avtp/openavb_avtp.c:562

2 openavbAvtpRx (pv=0x7fffe0000950) at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/avtp/openavb_avtp.c:656

3 0x000000000041b3fc in listenerDoStream (pTLState=0x664220)

at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/tl/openavb_listener.c:183

4 openavbTLRunListener (pTLState=pTLState@entry=0x664220)

at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/tl/openavb_listener.c:272

5 0x000000000041f78c in openavbTLThreadFn (pv=0x664220)

at /home/ethlab3/Downloads/Raman/Test_package_1.9/OpenAvnu/lib/avtp_pipeline/tl/openavb_tl_endpoint.c:108

6 0x00007ffff76b46ba in start_thread (arg=0x7ffff7f84700) at pthread_create.c:333

7 0x00007ffff5dd841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[tengandreaxu]

Hello Raman, I tried the following settings:

avtp_pipeline.mk AVB_FEATURE_ENDPOINT ?= 1 IGB_LAUNCHTIME_ENABLED ?= 1 PLATFORM_TOOLCHAIN ?= x86_i210_linux

x86_i210_linux.cmake set ( AVB_FEATURE_IGB 1) set ( AVB_FEATURE_PCAP 1)

With the following Testbed:

PC1 [I210] <--> Motu AVB switch <--> [I210] PC2

And as you stated before I can see "Segmentation Fault" (core dumped) on the Listener side even with one single stream. What have you done in order to make it work for a single stream?

Best,

[Raman-Jamloki]

There are issues in using pcap library in multi threaded environment.

[Raman-Jamloki]

crashfixes.zip

[Raman-Jamloki]

It may help

[Raman-Jamloki]

Let me know if you find some thing else.

[tengandreaxu]

Thank you @Raman-Jamloki I will back in Monday and let you know :) By the way do you see any "AVTP Mismatch" using either pcap or igb implementation?

[Raman-Jamloki]

Yes, and it does not recover. My observation is that once the streaming start with IGB based implementation(with the crash fix changes) it run smoothly (have tested it running for 3 days continuously with 3 streams). Crash happens at startup during talker/listener advertise and declaration. IGB based implementation is using pcap only and crash always points to pcap raw sock layer(checked with gdb). One thing I have not tried that use gptp code with ARCH=I210 when using IGB based implementation. It should not make any difference but can be checked once.

[tengandreaxu]

I have tried building gptp with ARCH=I210 with pcap implementation, I can still see random AVTP mismatch happening, like 100 frames have a mismatch over one milion.

[tengandreaxu]

Are you running on 1Gb/s bandwidth?

[Raman-Jamloki]

yes, 1Gb/s right now . I have to check on 100 Mbps too. Anything on crash. Did those fixes help?

[tengandreaxu]

I didn't implement your fix yet, I thought was to handle multiple streams. Anyway, with which brand of switch are you working on ? I just want to be sure that the AVTP miscmatch it's not only Motu Switch related problem.

[Raman-Jamloki]

I have netgear

[sofiamorseletto]

Hello @Raman-Jamloki

I have been dealing with the same issue as you but with one stream. I get a Segmentation Fault on the listener once the talker starts transmitting.

I tried implementing your crash fixes, but the result is the same.

Maybe I am implementing them wrong, these are the steps I followed:

1. On _/.../OpenAvnu/lib/avtp_pipeline/platform/Linux/rawsock/pcaprawsock.c I added these lines and changed the following function:

#define CRASH_FIXES

#ifdef CRASH_FIXES
static pthread_mutex_t cs_mutex = PTHREAD_MUTEX_INITIALIZER;
#endif
// Setup the rawsock to receive multicast packets
bool pcapRawsockRxMulticast(void *pvRawsock, bool add_membership, const U8 addr[ETH_ALEN])
{
pcap_rawsock_t *rawsock = (pcap_rawsock_t*)pvRawsock;

struct bpf_program comp_filter_exp;
char filter_exp[60];

sprintf(filter_exp, "ether dst %02x:%02x:%02x:%02x:%02x:%02x", addr[0], addr[1], addr[2], addr[3], addr[4], addr[5]);

AVB_LOGF_DEBUG("%s %d %s", __func__, (int)add_membership, filter_exp);
#ifdef CRASH_FIXES
pthread_mutex_lock(&cs_mutex);
#endif
if (pcap_compile(rawsock->handle, &comp_filter_exp, filter_exp, 0, PCAP_NETMASK_UNKNOWN) < 0) {
AVB_LOGF_ERROR("Could not parse filter %s: %s", filter_exp, pcap_geterr(rawsock->handle));
return false;
}

if (pcap_setfilter(rawsock->handle, &comp_filter_exp) < 0) {
AVB_LOGF_ERROR("Could not install filter %s: %s", filter_exp, pcap_geterr(rawsock->handle));
return false;
}
#ifdef CRASH_FIXES
pthread_mutex_unlock(&cs_mutex);
#endif
return true;
}

2. And in _/.../OpenAvnu/lib/avtp_pipeline/platform/Linux/rawsock/igbrawsock.c I changed the following function:

void *igbRawsockOpen(igb_rawsock_t* rawsock, const char *ifname, bool rx_mode, bool tx_mode, U16 ethertype, U32 frame_size, U32 num_frames)
{
AVB_TRACE_ENTRY(AVB_TRACE_RAWSOCK);
#ifdef CRASH_FIXES
pcap_rawsock_t *rawsockPcap = calloc(1, sizeof(pcap_rawsock_t));
#endif
if (!pcapRawsockOpen((pcap_rawsock_t*)rawsockPcap, ifname, rx_mode,
            tx_mode, ethertype, frame_size, num_frames))
{
AVB_TRACE_EXIT(AVB_TRACE_RAWSOCK);
return NULL;
}
#ifdef CRASH_FIXES
memcpy(&rawsock->base, &rawsockPcap->base, sizeof(base_rawsock_t));
rawsock->handle = rawsockPcap->handle;
#endif
----------
--------
}

This is the console output:

Starting program: /home/apu3/pasar3/OpenAvnu/lib/avtp_pipeline/build/bin/openavb_harness mpeg2ts_file_listener.ini
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7fc6700 (LWP 1443)]
[New Thread 0x7ffff6556700 (LWP 1444)]
Opening: mpeg2ts_file_listener.ini
Configuring: mpeg2ts_file_listener.ini
[New Thread 0x7ffff7fb5700 (LWP 1445)]
Starting: mpeg2ts_file_listener.ini
[New Thread 0x7ffff7fa4700 (LWP 1446)]
attaching to 0000:03:00.0
attach_tx failed! (Unknown error -12)
attaching to 0000:02:00.0
attach_tx failed! (Unknown error -12)
attaching to 0000:01:00.0
[New Thread 0x7ffff5d55700 (LWP 1447)]
[New Thread 0x7ffff5554700 (LWP 1448)]
[New Thread 0x7ffff4b41700 (LWP 1449)]
[582368:836283896 OPENAVB osalTime] INFO: local_time = 582368833541395
[582368:836310827 OPENAVB osalTime] INFO: ml_phoffset = -5, ls_phoffset = -2816
[582368:836362189 OPENAVB osalTime] INFO: ml_freqffset = 1.000000, ls_freqoffset = 1.000000
[582368:836400055 OPENAVB Endpoint] INFO: AVTP Pipeline: 0.1.3 (Development)
[582368:837361330 OPENAVB Talker / Listener] INFO: AVTP Pipeline: 0.1.3 (Development)
[582368:837682119 OPENAVB Talker / Listener] INFO: Looking up symbol for function: openavbMapMpeg2tsInitialize
[582368:837716526 OPENAVB Talker / Listener] INFO: Looking up symbol for function: openavbIntfMpeg2tsFileInitialize
[582368:847802642 OPENAVB HAL Ethernet] INFO: TX buffers: 40
[582368:847815928 OPENAVB HAL Ethernet] INFO: IGB launch time feature is ENABLED
[582368:877486668 OPENAVB Endpoint SRP] INFO: detected domain Class A PRIO=3 VID=0002...
[582368:877500670 OPENAVB Endpoint SRP] INFO: detected domain Class B PRIO=2 VID=0002...
[582368:878946828 OPENAVB Listener] INFO: Attach 00:0d:b9:4d:14:c4/1
[582368:882345406 OPENAVB Endpoint Shaper] INFO: Shaper daemon available
[582371:134943951 OPENAVB Endpoint MAAP] INFO: MAAP address range 1 acquired:  0x91e0f000ee9b-0x91e0f000eeba (Size 32)
[582394:021049513 OPENAVB Endpoint] INFO: SRP listener callback uid=1: tlkrDecl=1
[582394:021449050 OPENAVB Listener] INFO: Starting stream: 00:0d:b9:4d:14:c4/1
[582394:021604030 OPENAVB Raw Socket] INFO: Using *igb* implementation
[582394:230808881 OPENAVB Endpoint] INFO: SRP listener callback uid=1: tlkrDecl=1

Thread 5 "openavb_harness" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fa4700 (LWP 1446)]
0x00000000004279ef in pcapRawsockGetRxFrame ()

And this is the GDB trace:

#0  0x00000000004279ef in pcapRawsockGetRxFrame ()
#1  0x00000000004299d4 in openavbAvtpRx ()
#2  0x000000000041ae2c in openavbTLRunListener ()
#3  0x000000000042090c in openavbTLThreadFn ()
#4  0x00007ffff76b46ba in start_thread (arg=0x7ffff7fa4700) at pthread_create.c:333
#5  0x00007ffff6a9341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Any idea of what could be the issue?

Any help is appreciated. Thank you. Best regards, Sofia

[JakubMielczarek81]

Hi Sofia,

It's hard to tell from this GDB trace, could be some issue with pcap. If you just want to run the listener (and not debug this issue), you can try to change to raw socket type used by listener.

In you .ini file try this setting: ifname = ring:

This will select the "ring" raw socket implementation.

There are others: igb (default), pcap, simple.

Best regards, Jakub

[sofiamorseletto]

Hello, Jakub.

Thank you for the advice. However, I've been told that the best implementation for my configuration would be the igb one (#812 ), which is the one that is giving me a Segmentation Fault.

I think I'm not being able to run gdb in an appropiate way though. I'm getting the following message when I run sudo gdb --args openavb_harness mpeg2ts_file_talker.ini:

Reading symbols from openavb_harness...(no debugging symbols found)...done.

Do you know where exactly in the compilation process should I add the corresponding flags for gdb to work?

Thanks. Best, Sofia

[JakubMielczarek81]

Hi Sofia,

I don't remember, but you can try building "Debug" variant.

Actually "igb" socket (as far as I know, may miss something) on listener does not give you anything extra, it uses pcap library underneath. "igb" implementation is good for talkers, as it allows to use all features of the i210.

Best regards, Jakub

[Raman-Jamloki]

Search for CMAKE_BUILD_TYPE. It needs to be set to RelWithDebInfo before compilation. File name I don't remember

Line number 4 set(CMAKE_BUILD_TYPE RelWithDebInfo)

On Tue, Jun 25, 2019, 8:28 PM JakubMielczarek81 notifications@github.com wrote:

Hi Sofia,

I don't remember, but you can try building "Debug" variant.

Actually "igb" socket (as far as I know, may miss something) on listener does not give you anything extra, it uses pcap library underneath. "igb" implementation is good for talkers, as it allows to use all features of the i210.

Best regards, Jakub

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AVnu/OpenAvnu/issues/826?email_source=notifications&email_token=AJKY3IB4DCDYTWI6QLH2BR3P4IXCFA5CNFSM4FRY6B72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYQRCMI#issuecomment-505483569, or mute the thread https://github.com/notifications/unsubscribe-auth/AJKY3IAEBXHI4ZI4COLMD7TP4IXCFANCNFSM4FRY6B7Q .