Restore MAC from NC-SI

[bluecmd]

On bootup u-bmc should read MAC address from NC-SI OEM commands and configure the interface accordingly.

[bluecmd]

There is activity on the kernel mailinglist from the good folks @shenki and @sammj. Hopefully those patches will land in OpenBMC's kernel tree soon which means that I can get some idea on how to do OEM NC-SI. Still need to figure out the OEM commands, Mellanox do not want to provide the documentation but there are tools out there that we could study if they are not part of the patches mentioned.

[bluecmd]

Applying the following patches:

net-next-net-ncsi-Add-NCSI-OEM-command-support.diff
v2-1-2-net-ncsi-Add-NCSI-Broadcom-OEM-command.diff
net-next-v3-2-2-net-ncsi-Add-NCSI-Mellanox-OEM-command.diff
net-next-v7-net-ncsi-Extend-NC-SI-Netlink-interface-to-allow-user-space-to-send-NC-SI-command.diff

(full diff: https://gist.github.com/bluecmd/8d17d8716a5bffb752abbee0317576a6)

and enabling NCSI_OEM_CMD_GET_MAC=y.

Still get Generated random MAC address. Will investigate.

[bluecmd]

Adding some debug prints for sending/receiving GMA it can be seen that the command is sent but no good reply is received.

<6>[   15.446453] ncsi_dev_state_config_sp
<6>[   15.446807] ncsi_dev_state_config_cis
<6>[   15.447126] ncsi_dev_state_config_oem_gma
<6>[   15.447214] MLX OEM GMA sent: 0, 1b
<4>[   15.447405] ftgmac100 1e660000.ethernet eth0: NCSI: 'bad' packet ignored for type 0xd0
<6>[   15.463328] ncsi_dev_state_config_xx: 772
<4>[   15.463623] ftgmac100 1e660000.ethernet eth0: NCSI: 'bad' packet ignored for type 0x8b
<6>[   15.479554] ncsi_dev_state_config_xx: 772
<6>[   15.479590] ncsi_dev_state_config_xx: 773
<6>[   15.479615] ncsi_dev_state_config_xx: 774
<6>[   15.479962] ncsi_dev_state_config_xx: 775
<6>[   15.480336] ncsi_dev_state_config_xx: 776
<6>[   15.480770] ncsi_dev_state_config_xx: 778
<6>[   15.481135] ncsi_dev_state_config_xx: 779
<6>[   15.481489] ncsi_dev_state_config_xx: 781
<6>[   15.546083] ncsi_inet6addr_event EGMF mode
<6>[   15.546103] ncsi_inet6addr_event called

[bluecmd]

More debugging showed that other BMCs uses a 7 byte payload with all zeros instead of the other patch. The reason code (when using len 7 and no param) is also set to 2 which is "Parameter is invalid, Unsupported, or Out-of-range".

<3>[   12.264813] ftgmac100 1e660000.ethernet eth0: NCSI: Handler for packet type 0x82 returned -19
<6>[   15.011070] ncsi_inet6addr_event EGMF mode
<6>[   15.011089] ncsi_inet6addr_event called
<7>[   15.377395] ftgmac100 1e660000.ethernet eth0: NCSI: configuring channel 0
<6>[   15.377418] ncsi_dev_state_config_sp
<6>[   15.377729] ncsi_dev_state_config_cis
<6>[   15.378201] ncsi_dev_state_config_oem_gma
<6>[   15.378318] MLX OEM GMA sent: 00 00 81 19 00
<7>[   15.378509] ftgmac100 1e660000.ethernet eth0: NCSI: non zero response/reason code (0, 2)
<4>[   15.378549] ftgmac100 1e660000.ethernet eth0: NCSI: 'bad' packet ignored for type 0xd0
<6>[   15.394477] ncsi_dev_state_config_xx: 772
<7>[   15.394778] ftgmac100 1e660000.ethernet eth0: NCSI: non zero response/reason code (0, 2823)
<4>[   15.394821] ftgmac100 1e660000.ethernet eth0: NCSI: 'bad' packet ignored for type 0x8b
<6>[   15.410741] ncsi_dev_state_config_xx: 772
<7>[   15.411028] ftgmac100 1e660000.ethernet eth0: NCSI: non zero response/reason code (0, 2823)
<4>[   15.411070] ftgmac100 1e660000.ethernet eth0: NCSI: 'bad' packet ignored for type 0x8b
<6>[   15.426982] ncsi_dev_state_config_xx: 772
<6>[   15.427017] ncsi_dev_state_config_xx: 773
<6>[   15.427039] ncsi_dev_state_config_xx: 774
<6>[   15.427366] ncsi_dev_state_config_xx: 775

[bluecmd]

Reference PCAP dumped from stock BMC when setting up the MAC address: https://storage.googleapis.com/bluecmd/f06-mlx-ncsi-init.pcap Mac is E4:1D:2D:FC:2B:50 as can be seen in frame 28.

[bluecmd]

Basic NCSI wireshark dissector to read the above: https://github.com/bluecmd/ncsi-dissector/blob/master/packet-ncsi.c

[bluecmd]

Hacking the kernel to send the GMA command over and over again I was able to capture and compare.

The changes that in the end made a difference:

1. Min frame size set to 12 bytes to mimic the 46 bytes seen from stock BMC
2. Size 7, payload all zero
3. Disable checksum generation.

3 seems to be paramount and the only change I know is really needed.

I'll try with the original upstream patch again now with the checksum disabled.

[bluecmd]

Original patch with the 1b and 8 length does not work (response 0x3, reason 0x7fff). PCAP here: https://storage.googleapis.com/bluecmd/ncsi-u-bmc-with-1b.pcap

[bluecmd]

This patch on top of the above works for CX341 (but it doesn't do the +2 MAC yet so there is a collision).

diff --git a/net/ncsi/internal.h b/net/ncsi/internal.h
index 71670d3270f3..6dc306aab661 100644
--- a/net/ncsi/internal.h
+++ b/net/ncsi/internal.h
@@ -75,13 +75,12 @@ enum {
 #define NCSI_OEM_BCM_CMD_GMA            0x01   /* CMD ID for Get MAC */
 /* Mellanox specific OEM Command */
 #define NCSI_OEM_MLX_CMD_GMA            0x00   /* CMD ID for Get MAC */
-#define NCSI_OEM_MLX_CMD_GMA_PARAM      0x1b   /* Parameter for GMA  */
 /* OEM Command payload lengths*/
 #define NCSI_OEM_BCM_CMD_GMA_LEN        12
-#define NCSI_OEM_MLX_CMD_GMA_LEN        8
+#define NCSI_OEM_MLX_CMD_GMA_LEN        7
 /* Mac address offset in OEM response */
 #define BCM_MAC_ADDR_OFFSET             28
-#define MLX_MAC_ADDR_OFFSET             8
+#define MLX_MAC_ADDR_OFFSET             4

 struct ncsi_channel_version {
diff --git a/net/ncsi/ncsi-cmd.c b/net/ncsi/ncsi-cmd.c
index 356af474e43c..9b7ebc6fe20c 100644
--- a/net/ncsi/ncsi-cmd.c
+++ b/net/ncsi/ncsi-cmd.c
@@ -54,6 +54,10 @@ static void ncsi_cmd_build_header(struct ncsi_pkt_hdr *h,
        h->reserved1[0] = 0;
        h->reserved1[1] = 0;

+       if (nca->type == NCSI_PKT_CMD_OEM) {
+               return;
+       }
+
        /* Fill with calculated checksum */
        checksum = ncsi_calculate_checksum((unsigned char *)h,
                                           sizeof(*h) + nca->payload);
diff --git a/net/ncsi/ncsi-manage.c b/net/ncsi/ncsi-manage.c
index 961ff26dfd4c..3def8531cf89 100644
--- a/net/ncsi/ncsi-manage.c
+++ b/net/ncsi/ncsi-manage.c
@@ -684,7 +684,6 @@ static void ncsi_oem_gma_handler_mlx(struct ncsi_cmd_arg *nca)
        memset(data, 0, NCSI_OEM_MLX_CMD_GMA_LEN);
        *(unsigned int *)data = ntohl(NCSI_OEM_MFR_MLX_ID);
        data[5] = NCSI_OEM_MLX_CMD_GMA;
-       data[6] = NCSI_OEM_MLX_CMD_GMA_PARAM;

        nca->data = data;

diff --git a/net/ncsi/ncsi-rsp.c b/net/ncsi/ncsi-rsp.c
index 56145f9fc92d..8818691b9c7c 100644
--- a/net/ncsi/ncsi-rsp.c
+++ b/net/ncsi/ncsi-rsp.c
@@ -644,8 +644,7 @@ static int ncsi_rsp_handler_oem_mlx(struct ncsi_request *nr)
        rsp = (struct ncsi_rsp_oem_pkt *)skb_network_header(nr->rsp);
        mlx = (struct ncsi_rsp_oem_mlx_pkt *)(rsp->data);

-       if (mlx->cmd == NCSI_OEM_MLX_CMD_GMA &&
-           mlx->param == NCSI_OEM_MLX_CMD_GMA_PARAM)
+       if (mlx->cmd == NCSI_OEM_MLX_CMD_GMA)
                return ncsi_rsp_handler_oem_mlx_gma(nr);
        return 0;
 }

Confirmed working

[bluecmd]

IPv6 link local addresses seems to be sometimes correct, sometimes bad. Need to investigate why they are not updated when the MAC changes.