search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
727 stars 1.39k forks source link

Failed to load ipd.ko on 202012-innovium #17566

Open konradkreciwilk opened 9 months ago

konradkreciwilk commented 9 months ago

Description

Failed to load innovium-platform-driver (ipd.ko) for latest branch 202012-innovium.

Steps to reproduce the issue:

modprobe ipd

modprobe: ERROR: could not insert 'ipd': Invalid argument

Describe the results you received:

dpkg -l innovium-platform-driver

Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-========================-============-============-========================= ii innovium-platform-driver 0.107.9 amd64 Innovium Platform Driver

dmesg

[ 367.205689] ipd: disagrees about version of symbol eth_mac_addr [ 367.205692] ipd: Unknown symbol eth_mac_addr (err -22) [ 367.205702] ipd: disagrees about version of symbol skb_put [ 367.205704] ipd: Unknown symbol skb_put (err -22) [ 367.205723] ipd: disagrees about version of symbol unregister_netdev [ 367.205725] ipd: Unknown symbol unregister_netdev (err -22) [ 367.205755] ipd: disagrees about version of symbol skb_add_rx_frag [ 367.205757] ipd: Unknown symbol skb_add_rx_frag (err -22) [ 367.205814] ipd: disagrees about version of symbol ether_setup [ 367.205816] ipd: Unknown symbol ether_setup (err -22) [ 367.205832] ipd: disagrees about version of symbol eth_type_trans [ 367.205834] ipd: Unknown symbol eth_type_trans (err -22) [ 367.205852] ipd: disagrees about version of symbol napi_complete_done [ 367.205854] ipd: Unknown symbol napi_complete_done (err -22) [ 367.205859] ipd: disagrees about version of symbol alloc_netdev_mqs [ 367.205861] ipd: Unknown symbol alloc_netdev_mqs (err -22) [ 367.205866] ipd: disagrees about version of symbol kfree_skb [ 367.205868] ipd: Unknown symbol kfree_skb (err -22) [ 367.205887] ipd: disagrees about version of symbol napi_schedule [ 367.205889] ipd: Unknown symbol __napi_schedule (err -22) [ 367.205906] ipd: disagrees about version of symbol alloc_skb [ 367.205908] ipd: Unknown symbol alloc_skb (err -22) [ 367.205913] ipd: disagrees about version of symbol netif_device_detach [ 367.205915] ipd: Unknown symbol netif_device_detach (err -22) [ 367.205944] ipd: disagrees about version of symbol dev_open [ 367.205946] ipd: Unknown symbol dev_open (err -22) [ 367.205960] ipd: disagrees about version of symbol skb_pull [ 367.205962] ipd: Unknown symbol skb_pull (err -22) [ 367.205978] ipd: disagrees about version of symbol netif_napi_add [ 367.205980] ipd: Unknown symbol netif_napi_add (err -22) [ 367.205989] ipd: disagrees about version of symbol dev_close [ 367.205991] ipd: Unknown symbol dev_close (err -22) [ 367.205995] ipd: disagrees about version of symbol skb_push [ 367.205997] ipd: Unknown symbol skb_push (err -22) [ 367.206004] ipd: disagrees about version of symbol netif_receive_skb [ 367.206006] ipd: Unknown symbol netif_receive_skb (err -22) [ 367.206014] ipd: disagrees about version of symbol register_netdev [ 367.206016] ipd: Unknown symbol register_netdev (err -22) [ 367.206025] ipd: disagrees about version of symbol free_netdev [ 367.206027] ipd: Unknown symbol free_netdev (err -22) [ 367.206059] ipd: disagrees about version of symbol netif_napi_del [ 367.206061] ipd: Unknown symbol netif_napi_del (err -22) [ 367.206083] ipd: disagrees about version of symbol __dev_kfree_skb_any [ 367.206085] ipd: Unknown symbol dev_kfree_skb_any (err -22) [ 367.206092] ipd: disagrees about version of symbol netif_carrier_off [ 367.206094] ipd: Unknown symbol netif_carrier_off (err -22) [ 367.206100] ipd: disagrees about version of symbol skb_copy [ 367.206102] ipd: Unknown symbol skb_copy (err -22) [ 367.206105] ipd: disagrees about version of symbol netif_carrier_on [ 367.206107] ipd: Unknown symbol netif_carrier_on (err -22) [ 367.206124] ipd: disagrees about version of symbol napi_schedule_prep [ 367.206126] ipd: Unknown symbol napi_schedule_prep (err -22) [ 367.206133] ipd: disagrees about version of symbol napi_disable

Output of show version:

SONiC Software Version: SONiC.202012.432598-298401795 Distribution: Debian 10.13 Kernel: 4.19.0-12-2-amd64 Build commit: 298401795 Build date: Wed Dec 13 10:49:47 UTC 2023 Built by: AzDevOps@vmss-soni002NH3

Platform: x86_64-netberg_aurora_615-r0 HwSKU: None ASIC: innovium ASIC Count: 1 Serial Number: Uptime: 05:51:16 up 6 min, 1 user, load average: 0.93, 0.83, 0.43

Docker images: REPOSITORY TAG IMAGE ID SIZE docker-fpm-frr 202012.432598-298401795 40557db09eba 361MB docker-fpm-frr latest 40557db09eba 361MB docker-sonic-mgmt-framework 202012.432598-298401795 4ce18e60d17c 658MB docker-sonic-mgmt-framework latest 4ce18e60d17c 658MB docker-sflow 202012.432598-298401795 7adcc0ac1ece 344MB docker-sflow latest 7adcc0ac1ece 344MB docker-orchagent 202012.432598-298401795 47e39095a75e 361MB docker-orchagent latest 47e39095a75e 361MB docker-teamd 202012.432598-298401795 5fe59d2e0383 343MB docker-teamd latest 5fe59d2e0383 343MB docker-nat 202012.432598-298401795 d8f7f4a41cf3 345MB docker-nat latest d8f7f4a41cf3 345MB docker-sonic-telemetry 202012.432598-298401795 2a901a3efac3 421MB docker-sonic-telemetry latest 2a901a3efac3 421MB docker-platform-monitor 202012.432598-298401795 aee1320e6e61 516MB docker-platform-monitor latest aee1320e6e61 516MB docker-snmp 202012.432598-298401795 07373a1cf39b 374MB docker-snmp latest 07373a1cf39b 374MB docker-syncd-invm 202012.432598-298401795 74437367f346 429MB docker-syncd-invm latest 74437367f346 429MB docker-lldp 202012.432598-298401795 5f6b265e3538 371MB docker-lldp latest 5f6b265e3538 371MB docker-database 202012.432598-298401795 19549cc99626 331MB docker-database latest 19549cc99626 331MB docker-dhcp-relay 202012.432598-298401795 5f830c5c8eb1 344MB docker-dhcp-relay latest 5f830c5c8eb1 344MB docker-mux 202012.432598-298401795 23f23c7f259b 384MB docker-mux latest 23f23c7f259b 384MB docker-router-advertiser 202012.432598-298401795 556f8e39244b 331MB docker-router-advertiser latest 556f8e39244b 331MB

judyjoseph commented 8 months ago

Please could you triage further on this kernel module issue?

kishoreatreya commented 8 months ago

@ggopi-marvell and @radha-danda please take a look at this.

Keshavg-marvell commented 8 months ago

Looks like this is caused due to recent change in sonic202012 kernel code. We are actively working on it resolve this.

Keshavg-marvell commented 6 months ago

@konradkreciwilk , New debians are released. please verify and close this issue

konradkreciwilk commented 5 months ago

@Keshavg-marvell, ipd.ko for branch-innovium is loaded correctly now, thank you !

Currently I observe issue with syncd container, when swss starts, after a few seconds syncd reports SAI FAILURE

root@sonic:/# syncd_start.sh

ModuleNotFoundError: No module named 'ifcscshell' ERROR: IVM-SDK: [SWITCH] ii_node_cli_invoke_impl: Switch cli failed to load module ifcscshell ERROR: IVM-SDK: [ROUTE] [z17227712cd:0] Route HW slot limit exceeded. rc: n/a.n/a.n/a.n/a.IFCS_RESOURCE_FULL, slot_size: 1, hw_slot_usage: 1, route__ipv6_prefix_max_slots: 0, route__ipv6_prefix_2x_max_slots: 0 ERROR: IVM-SDK: [ROUTE] [z69e47778e3:0] STA route add failed, rc:13 ERROR: IVM-SDK: [ROUTE] [ii_k2_route_ilpm_validate_impl][0] RouteAdd:Error during sta route add. Ret 13! ERROR: IVM-SDK: [ROUTE] [ii_route_entry_create][0] ROUTE_ENTRY Failed to create PD Objects rc 13 ERROR: IVM-SAI: Route: Create Route: [rc = SAI_STATUS_TABLE_FULL]: Failed to create route in IFCS ERROR: IVM-SAI: route_entry: Create route_entry: [rc = SAI_STATUS_TABLE_FULL]: Create Failed

For compile innovium debians, are you use latest commits from branch 202012 https://github.com/sonic-net/sonic-sairedis/tree/202012 https://github.com/sonic-net/sonic-swss/tree/202012 https://github.com/sonic-net/sonic-swss-common/tree/202012 ?