search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
741 stars 1.44k forks source link

[logs][kdump] Logs show "kdump-tools[635]: failed!" on boot #9629

Open alexrallen opened 2 years ago

alexrallen commented 2 years ago

Description

Receiving error log during switch boot.

Steps to reproduce the issue:

  1. Install SONiC latest master via ONIE
  2. Boot switch and check log

Describe the results you received:

Dec 22 04:26:05 sonic kdump-tools[610]: no crashkernel= parameter in the kernel cmdline ...
Dec 22 04:26:05 sonic kdump-tools[610]: no crashkernel= parameter in the kernel cmdline ...
Dec 22 04:26:05 sonic kdump-tools[635]:  failed!
Dec 22 04:26:05 sonic kdump-tools[635]:  failed!

Describe the results you expected:

No error log.

Output of show version:

SONiC Software Version: SONiC.master.244-3aec72879_Internal
Distribution: Debian 11.2
Kernel: 5.10.0-8-2-amd64
Build commit: 3aec72879
Build date: Wed Dec 22 08:30:03 UTC 2021
Built by: sw-r2d2-bot@r-build-sonic-ci02-241

Platform: x86_64-mlnx_msn3420-r0
HwSKU: ACS-MSN3420
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2019X13878
Model Number: MSN3420-CB2FO
Hardware Revision: A1
Uptime: 22:05:10 up  5:42,  5 users,  load average: 1.11, 1.01, 0.78

Docker images:
REPOSITORY                                         TAG                             IMAGE ID       SIZE
docker-platform-monitor                            latest                          3179b3bfa4a2   809MB
docker-platform-monitor                            master.244-3aec72879_Internal   3179b3bfa4a2   809MB
docker-teamd                                       latest                          2f47d005bb95   436MB
docker-teamd                                       master.244-3aec72879_Internal   2f47d005bb95   436MB
docker-syncd-mlnx                                  latest                          cc4d31e33915   1.01GB
docker-syncd-mlnx                                  master.244-3aec72879_Internal   cc4d31e33915   1.01GB
docker-orchagent                                   latest                          5c891ff8f214   455MB
docker-orchagent                                   master.244-3aec72879_Internal   5c891ff8f214   455MB
docker-dhcp-relay                                  latest                          c646bc83cc8c   436MB
docker-sonic-telemetry                             latest                          606f26ae194f   511MB
docker-sonic-telemetry                             master.244-3aec72879_Internal   606f26ae194f   511MB
docker-sonic-mgmt-framework                        latest                          7c86fd946748   578MB
docker-sonic-mgmt-framework                        master.244-3aec72879_Internal   7c86fd946748   578MB
docker-snmp                                        latest                          546605aaf36c   465MB
docker-snmp                                        master.244-3aec72879_Internal   546605aaf36c   465MB
docker-sflow                                       latest                          68f7233b21a9   436MB
docker-sflow                                       master.244-3aec72879_Internal   68f7233b21a9   436MB
docker-router-advertiser                           latest                          dafc3d1fb775   423MB
docker-router-advertiser                           master.244-3aec72879_Internal   dafc3d1fb775   423MB
docker-nat                                         latest                          47bf92fce979   438MB
docker-nat                                         master.244-3aec72879_Internal   47bf92fce979   438MB
docker-mux                                         latest                          b68969e05d75   475MB
docker-mux                                         master.244-3aec72879_Internal   b68969e05d75   475MB
docker-macsec                                      latest                          acb05bec6969   439MB
docker-macsec                                      master.244-3aec72879_Internal   acb05bec6969   439MB
docker-lldp                                        latest                          85f1ea2eb1a0   463MB
docker-lldp                                        master.244-3aec72879_Internal   85f1ea2eb1a0   463MB
docker-fpm-frr                                     latest                          ebb395f4d389   454MB
docker-fpm-frr                                     master.244-3aec72879_Internal   ebb395f4d389   454MB
docker-database                                    latest                          89d24c1c4f64   423MB
docker-database                                    master.244-3aec72879_Internal   89d24c1c4f64   423MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.0.0-master-internal-18        2290464c0e59   468MB
harbor.mellanox.com/sonic/cpu-report               10.0.0                          5314b41a2a5e   413MB
zhangyanzhao commented 2 years ago

BRCM will help to take a look.

dgsudharsan commented 2 years ago

@adyeung any update on this issue?

Yuval-Mellanox commented 2 years ago

@zhangyanzhao any update on this issue?

rajendra-dendukuri commented 2 years ago

I checked SONiC.master.129888-7f7800622 broadcom image and it appears that the kdump-tools package itself is not installed. So I am seeing other errors and not the ones mentioned here. I need to fix the build first before I can proceed to fix this error.

It is likely that during first boot, the grub.cfg doesn't have the crashkernel value set. As a result, when kdump tries to enable it we may be seeing the reported error. I will confirm this soon.