[ZTP] ZTP runs dhclient on inband interfaces while config DB json file exists

liorghub commented 11 months ago

Description

ZTP ran dhclient on inband interfaces on init although config DB file exists under /etc/sonic/config_db.json

Steps to reproduce the issue:

Perform onie install to version from 202305 branch (stated below).
Copy to switch the attached config_db.json
Reboot the switch (regular reboot)

Describe the results you received:

On init, ZTP runs dhclient on inband interfaces. The following errors appear in log:

r-spider-05 INFO networking[690]: error: Ethernet0: cmd '/sbin/dhclient -pf /run/dhclient.Ethernet0.pid -lf /var/lib/dhcp/dhclient.Ethernet0.leases Ethernet0 -nw' failed: returned 1
r-spider-05 INFO networking[690]: error: Ethernet100: cmd '/sbin/dhclient -pf /run/dhclient.Ethernet100.pid -lf /var/lib/dhcp/dhclient.Ethernet100.leases Ethernet100 -nw' failed: returned 1
r-spider-05 INFO networking[690]: error: Ethernet104: cmd '/sbin/dhclient -pf /run/dhclient.Ethernet104.pid -lf /var/lib/dhcp/dhclient.Ethernet104.leases Ethernet104 -nw' failed: returned 1
r-spider-05 INFO networking[690]: error: Ethernet108: cmd '/sbin/dhclient -pf /run/dhclient.Ethernet108.pid -lf /var/lib/dhcp/dhclient.Ethernet108.leases Ethernet108 -nw' failed: returned 1
r-spider-05 INFO networking[690]: error: Ethernet112: cmd '/sbin/dhclient -pf /run/dhclient.Ethernet112.pid -lf /var/lib/dhcp/dhclient.Ethernet112.leases Ethernet112 -nw' failed: returned 1
r-spider-05 INFO networking[690]: error: Ethernet116: cmd '/sbin/dhclient -pf /run/dhclient.Ethernet116.pid -lf /var/lib/dhcp/dhclient.Ethernet116.leases Ethernet116 -nw' failed: returned 1

r-spider-05 INFO systemd[1]: networking.service: Sent signal SIGTERM to main process 281 (start-networkin) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 690 (python3) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 1050 (dhclient) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 2122 (dhclient) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 10863 (dhclient) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 10864 (dhclient) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 10869 (dhclient-script) on client request.
r-spider-05 INFO systemd[1]: networking.service: Sending signal SIGTERM to process 10875 (ztp) on client request.

Describe the results you expected:

dhclient should not run on inband interfaces if inband ZTP is not enabled.

Output of `show version`:

root@r-bulldog-04:/home/admin# show version

SONiC Software Version: SONiC.202305_RC.21-cbe2c014b_Internal
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: de69d986c
Build date: Thu Nov  9 14:06:00 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci03-242

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1752X06332
Model Number: MSN2100-CB2F
Hardware Revision: A1
Uptime: 16:54:18 up 3 min,  1 user,  load average: 2.95, 2.29, 0.99
Date: Thu 16 Nov 2023 16:54:18

Docker images:
REPOSITORY                                         TAG                               IMAGE ID       SIZE
docker-syncd-mlnx                                  202305_RC.21-cbe2c014b_Internal   306ff884607c   836MB
docker-syncd-mlnx                                  latest                            306ff884607c   836MB
docker-platform-monitor                            202305_RC.21-cbe2c014b_Internal   9f4aecb32eb6   828MB
docker-platform-monitor                            latest                            9f4aecb32eb6   828MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.6.0-202305-7                    0f4dca74c945   433MB
docker-orchagent                                   202305_RC.21-cbe2c014b_Internal   b85f8e9d2567   328MB
docker-orchagent                                   latest                            b85f8e9d2567   328MB
docker-fpm-frr                                     202305_RC.21-cbe2c014b_Internal   5c43c09c8ee9   348MB
docker-fpm-frr                                     latest                            5c43c09c8ee9   348MB
docker-nat                                         202305_RC.21-cbe2c014b_Internal   711cc90a809b   320MB
docker-nat                                         latest                            711cc90a809b   320MB
docker-sflow                                       202305_RC.21-cbe2c014b_Internal   892f60ec53a8   318MB
docker-sflow                                       latest                            892f60ec53a8   318MB
docker-teamd                                       202305_RC.21-cbe2c014b_Internal   ea8cc2c42c33   317MB
docker-teamd                                       latest                            ea8cc2c42c33   317MB
docker-macsec                                      latest                            97c101398305   319MB
docker-dhcp-relay                                  latest                            ce1df3511f73   307MB
docker-eventd                                      202305_RC.21-cbe2c014b_Internal   81fb3bce71f9   299MB
docker-eventd                                      latest                            81fb3bce71f9   299MB
docker-sonic-telemetry                             202305_RC.21-cbe2c014b_Internal   faa54c27d516   386MB
docker-sonic-telemetry                             latest                            faa54c27d516   386MB
docker-snmp                                        202305_RC.21-cbe2c014b_Internal   7911150e4f0a   338MB
docker-snmp                                        latest                            7911150e4f0a   338MB
docker-lldp                                        202305_RC.21-cbe2c014b_Internal   8682586a3df2   341MB
docker-lldp                                        latest                            8682586a3df2   341MB
docker-database                                    202305_RC.21-cbe2c014b_Internal   41fd7b88a05b   299MB
docker-database                                    latest                            41fd7b88a05b   299MB
docker-router-advertiser                           202305_RC.21-cbe2c014b_Internal   f3509bfdcdec   299MB
docker-router-advertiser                           latest                            f3509bfdcdec   299MB
docker-mux                                         202305_RC.21-cbe2c014b_Internal   654d73279524   348MB
docker-mux                                         latest                            654d73279524   348MB
docker-sonic-mgmt-framework                        202305_RC.21-cbe2c014b_Internal   ec4f523b0fea   415MB
docker-sonic-mgmt-framework                        latest                            ec4f523b0fea   415MB

root@r-bulldog-04:/home/admin#

Output of `show techsupport`:

[sysdump_test_check_errors_in_log_during_deploy_sonic_image.tar.gz](https://github.com/sonic-net/sonic-buildimage/files/13380388/sysdump_test_check_errors_in_log_during_deploy_sonic_image.tar.gz)

[config_db.json](https://github.com/sonic-net/sonic-buildimage/files/13380393/config_db.json)

Additional information you deem important (e.g. issue happens only occasionally):

arlakshm commented 11 months ago

Hi @adyeung, can Broadcom triage this issue

adyeung commented 11 months ago

@rajendra-dendukuri to help take a look

rajendra-dendukuri commented 11 months ago

@liorghub How did you disable in-band ztp? By default ZTP is performed on both out of band and inband interfaces. To disable in-band ztp, you need to perform:

Modify /host/ztp/ztp_cfg.json to contain "feat-inband" : false or
- Modify source code "sonic-ztp/src/usr/lib/python3/dist-packages/ztp/defaults.py" to have "feat-inband" : False, \"

liorghub commented 11 months ago

@rajendra-dendukuri We did not explictly disable inband ztp. however, config DB json file existed on the switch therefore we expect ztp not to run at all. I have updated the bug description.

sonic-net / sonic-buildimage