PTP daemon looses sync with master after setting up SO_TIMESTAMPING socket

[chris-kuhr]

Hi *,

I have a problem with the hw timestamping of incoming ethernet frames.

The gPTP daemon looses sync and never recovers. reboot required.

    Oct 02 09:03:27 srv3 gPTP_daemon[1746]: ERROR    : GPTP [09:03:27:812] Invalid TX
    Oct 02 09:03:27 srv3 gPTP_daemon[1746]: ERROR    : GPTP [09:03:27:812] Error (TX) timestamping PDelay request, error=-72
    Oct 02 09:03:28 srv3 gPTP_daemon[1746]: ERROR    : GPTP [09:03:28:235] Error (TX) timestamping PDelay Response (Retrying-0), error=-72

I have setup up a raw socket with the following socket option, as it was proposed in https://github.com/AVnu/OpenAvnu/issues/437

    int timestamp_flags = 0;
    timestamp_flags |= SOF_TIMESTAMPING_RX_HARDWARE;
    timestamp_flags |= SOF_TIMESTAMPING_SYS_HARDWARE;
    timestamp_flags |= SOF_TIMESTAMPING_RAW_HARDWARE;

    struct hwtstamp_config hwconfig;
    memset( &hwconfig, 0, sizeof( hwconfig ));
    hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
    hwconfig.tx_type = HWTSTAMP_TX_OFF;

    struct ifreq hwtstamp;
    memset((char*)&hwtstamp, 0, sizeof(struct ifreq));
    strncpy(hwtstamp.ifr_name, ifr.ifr_name, IFNAMSIZ-1);
    hwtstamp.ifr_data = (void *) &hwconfig;

    if( ioctl( *raw_transport_socket, SIOCSHWTSTAMP, &hwtstamp ) == -1 ) {
       fprintf(filepointer,  "[RAW TRANSPORT] ioctl timestamping failed %d %s \n", errno, strerror(errno));
    close(*raw_transport_socket);
    return RETURN_VALUE_FAILURE;
}

if (setsockopt(*raw_transport_socket, SOL_SOCKET, SO_TIMESTAMPING, &timestamp_flags, sizeof(timestamp_flags) ) == -1) {
    fprintf(filepointer,  "[RAW TRANSPORT] setsockopt timestamping failed %d %s \n", errno, strerror(errno));
    close(*raw_transport_socket);
    return RETURN_VALUE_FAILURE;
} else {
    fprintf(filepointer,  "[RAW TRANSPORT] Timestamp Socket \n");
}

...and try to read the ancillary data as follows:

    int wait_recv_ts( ieee1722_avtp_driver_state **ieee1722mc, struct sockaddr_in **si_other_avb, struct pollfd **avtp_transport_socket_fds )
    {
        socklen_t slen_avb = sizeof(struct sockaddr_in);
        char stream_packet[BUFLEN];
        struct iovec iov = { stream_packet, BUFLEN };
        struct msghdr msg = { (void*)((struct sockaddr *)(*si_other_avb)), slen_avb, &iov, 1, NULL, 0, 0 };

        int status = recvmsg((*avtp_transport_socket_fds)->fd, &msg, NULL);

        if (status < 0) {
            fprintf(filepointer, "Error recvmsg: %d %d %s\n", status, errno, strerror(errno));
            return -1;
        }

        if (status == 0) {
            fprintf(filepointer, "EOF\n");
            return -1;
        }
        if(
            ((*ieee1722mc)->streamid8[0] == (uint8_t) stream_packet[18]) &&
            ((*ieee1722mc)->streamid8[1] == (uint8_t) stream_packet[19]) &&
            ((*ieee1722mc)->streamid8[2] == (uint8_t) stream_packet[20]) &&
            ((*ieee1722mc)->streamid8[3] == (uint8_t) stream_packet[21]) &&
            ((*ieee1722mc)->streamid8[4] == (uint8_t) stream_packet[22]) &&
            ((*ieee1722mc)->streamid8[5] == (uint8_t) stream_packet[23]) &&
            ((*ieee1722mc)->streamid8[6] == (uint8_t) stream_packet[24]) &&
            ((*ieee1722mc)->streamid8[7] == (uint8_t) stream_packet[25])
        ){
            struct cmsghdr *cmsg = (struct cmsghdr *)malloc(sizeof(struct cmsghdr));
            cmsg = CMSG_FIRSTHDR(&msg);
            fprintf(filepointer, "stream packet! %d %d %d\n", cmsg->cmsg_len, cmsg->cmsg_level, cmsg->cmsg_type);
            for (cmsg = CMSG_FIRSTHDR(&msg); cmsg != NULL; cmsg = CMSG_NXTHDR(&msg, cmsg)){
                fprintf(filepointer, "stream packet!: %d %d\n", cmsg->cmsg_level, cmsg->cmsg_type);
                if (cmsg->cmsg_level != SOL_SOCKET)
                    continue;
                switch (cmsg->cmsg_type){
                    case SO_TIMESTAMPING:{
                            struct timespec* stamp = (struct timespec*)CMSG_DATA(cmsg); // timestamp is found
                            fprintf(filepointer, "Timestamp %ld sec %ld nanosec\n", stamp->tv_sec, stamp->tv_nsec);
                            return 0;
                        break;
                    }
                    default:
                            fprintf(filepointer, "no timestamp\n");fflush(filepointer);
                        break;
                }
            }
        }
        return -1;
    }

I cannot receive any avtp packet on this socket, since the bridge port is not AS Capable anymore.

Can someone tell me, what I am doing wrong?

Thanks! Best, Ck

[andrew-elder]

What devices are used in this setup and what firmware stack are they running?

[chris-kuhr]

Where can I find that info?

Here is my lspci output:

christoph@srv3 ~/sources/jack2.git $ lspci -vvv -s04:00.0
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
    Subsystem: Intel Corporation Ethernet Server Adapter I210-T1
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 35
    Region 0: Memory at fe200000 (32-bit, non-prefetchable) [size=1M]
    Region 3: Memory at fe300000 (32-bit, non-prefetchable) [size=16K]
    Expansion ROM at fe100000 [disabled] [size=1M]
    Capabilities: <access denied>
    Kernel driver in use: igb_avb
    Kernel modules: igb

[chris-kuhr]

How can I find the firmware stack version?

[andrew-elder]

PTP only works with 2 802.1AS capable devices on a network. I see you have an i210. What is the other 802.1AS capable device that it is connected to?

[chris-kuhr]

It's a Extreme Networks x440. my setup usually works fine.

[chris-kuhr]

any suggestions?

[andrew-elder]

My only suggestion would be to turn on more logging and do a wireshark capture. After that, more careful analysis will be required.

[chris-kuhr]

Ok, I have identified the problem.

It is mandatory to set

hwconfig.tx_type = HWTSTAMP_TX_ON;

otherwise the ptp daemon loses sync as.

[andrew-elder]

Great! It is not on by default?

[chris-kuhr]

Sorry, I think there is a misunderstanding...

I had an avtp listener socket configured for timestamping, not the ptp daemon. Thus, I only would want the Rx timestamps and configured the socket accordingly. Now I have tried these options since the ptp daemon works well with them.

It seems, that it affects the socket of the ptp daemon, when I open a new socket with SO_TIMESTAMPING in my application.

[andrew-elder]

Thanks for clarifying.