search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
730 stars 1.4k forks source link

make SONIC_BUILD_JOBS=4 all Fail #10468

Open zhengbiqing opened 2 years ago

zhengbiqing commented 2 years ago

Description

Steps to reproduce the issue:

  1. I follow "Usage" section as https://github.com/Azure/sonic-buildimage.
  2. there are some problems when "make init", but I fixed them. https://github.com/Azure/sonic-buildimage/issues/10357
  3. make configure PLATFORM=broadcom
  4. and run "make SONIC_BUILD_JOBS=4 all" many times, but there are various problems. for example, there are no output and no error for many hours. here are some output errors of some "make" commands:

A

[ 01 ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb-install ] [ 02 ] [ target/debs/bullseye/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ] [ 03 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] [ 04 ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] [ FAIL LOG START ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] [ REASON ] : target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb does not exist NON-EXISTENT PREREQUISITES: target/debs/bullseye/linux-headers-5.10.0-8-2-amd64_5.10.46-4_amd64.deb-install target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb-install [ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] /sonic/platform/broadcom/sonic-platform-modules-arista /sonic dpkg-buildpackage: error: cannot open file debian/changelog: No such file or directory [ FAIL LOG END ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] make: [slave.mk:520: target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb] Error 1 make: Waiting for unfinished jobs.... [ 01 ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb-install ] [ 02 ] [ target/debs/bullseye/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ] [ 03 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] [ 01 ] [ target/debs/bullseye/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ] [ 02 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] [ 01 ] [ target/debs/bullseye/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ] make[1]: Leaving directory '/home/zbq/sonic/sonic-buildimage'

B cmake .. make build-deb pushd debs mv libyang_1.0.73_amd64.deb /sonic/target/debs/bullseye/ mv libyang-dev_1.0.73_amd64.deb libyang-dbg_1.0.73_amd64.deb python3-yang_1.0.73_amd64.deb libyang-cpp_1.0.73_amd64.deb /sonic/target/debs/bullseye/ popd Cloning into 'libyang-1.0.73'... error: 4280 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed make[1]: [Makefile:10: /sonic/target/debs/bullseye/libyang_1.0.73_amd64.deb] Error 128 make[1]: Leaving directory '/sonic/src/libyang' [ FAIL LOG END ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb ] make: [slave.mk:481: target/debs/bullseye/libyang_1.0.73_amd64.deb] Error 1 [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] [ 02 ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] [ FAIL LOG START ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] [ REASON ] : target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb does not exist
[ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] make[1]: warning: -j2 forced in submake: resetting jobserver mode. make[1]: Entering directory '/sonic/src/radius/pam' Build freeradius-server v1.1.x pushd freeradius rm -rf freeradius-server git clone https://github.com/FreeRADIUS/freeradius-server.git popd pushd freeradius/freeradius-server git checkout -f 5f715dba4d2dbdb268bf60fcc656352274930941 Apply patch patch -p1 < ../patches/freeradius_configure.patch patch -p1 < ../patches/freeradius_2007-04-06.patch patch -p1 < ../patches/freeradius_libeap_deprecated_openssl_1_0.patch cp ../patches/config.sub . cp ../patches/config.guess . ./configure --disable-static --enable-libtool-lock make popd Obtain pam_radius rm -rf ./pam_radius git clone https://github.com/FreeRADIUS/pam_radius.git pushd ./pam_radius This is the latest commit with Radius MPL support git checkout -f 149c25df84cf5cd0e9addd9346699a9ca8fdddd2 cp -r ../debian . dpkg-buildpackage -rfakeroot -b -us -uc -nc popd mv libpam-radius-auth_1.4.1-1_amd64.deb /sonic/target/debs/bullseye/ /sonic/src/radius/pam/freeradius /sonic/src/radius/pam Cloning into 'freeradius-server'... error: 192 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed make[1]: [Makefile:8: /sonic/target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb] Error 128 make[1]: Leaving directory '/sonic/src/radius/pam' [ FAIL LOG END ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] make: [slave.mk:481: target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb] Error 1 [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] make[1]: Leaving directory '/home/zbq/sonic/sonic-buildimage'

C This is the latest commit with Radius MPL support git checkout -f 149c25df84cf5cd0e9addd9346699a9ca8fdddd2 cp -r ../debian . dpkg-buildpackage -rfakeroot -b -us -uc -nc popd mv libpam-radius-auth_1.4.1-1_amd64.deb /sonic/target/debs/bullseye/ /sonic/src/radius/pam/freeradius /sonic/src/radius/pam Cloning into 'freeradius-server'... fatal: unable to access 'https://github.com/FreeRADIUS/freeradius-server.git/': GnuTLS recv error (-54): Error in the pull function. make[1]: [Makefile:8: /sonic/target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb] Error 128 make[1]: Leaving directory '/sonic/src/radius/pam' [ FAIL LOG END ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] make: [slave.mk:481: target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb] Error 1 make: Waiting for unfinished jobs.... [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] [ 02 ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb ] [ 03 ] [ target/debs/bullseye/libnl-cli-3-200_3.5.0-1_amd64.deb-install ] [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] [ 02 ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb ] [ FAIL LOG START ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb ] [ REASON ] : target/debs/bullseye/libyang_1.0.73_amd64.deb does not exist
[ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] make[1]: warning: -j2 forced in submake: resetting jobserver mode. make[1]: Entering directory '/sonic/src/libyang' rm -fr ./libyang-1.0.73 git clone https://github.com/CESNET/libyang.git libyang-1.0.73 pushd libyang-1.0.73 git checkout tags/v1.0-r4 -b libyang Apply patch series stg init stg import -s ../patch/series mkdir build pushd build cmake .. make build-deb pushd debs mv libyang_1.0.73_amd64.deb /sonic/target/debs/bullseye/ mv libyang-dev_1.0.73_amd64.deb libyang-dbg_1.0.73_amd64.deb python3-yang_1.0.73_amd64.deb libyang-cpp_1.0.73_amd64.deb /sonic/target/debs/bullseye/ popd Cloning into 'libyang-1.0.73'... fatal: unable to access 'https://github.com/CESNET/libyang.git/': GnuTLS recv error (-54): Error in the pull function. make[1]: [Makefile:10: /sonic/target/debs/bullseye/libyang_1.0.73_amd64.deb] Error 128 make[1]: Leaving directory '/sonic/src/libyang' [ FAIL LOG END ] [ target/debs/bullseye/libyang_1.0.73_amd64.deb ] make: [slave.mk:481: target/debs/bullseye/libyang_1.0.73_amd64.deb] Error 1 [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] ^Cmake: [slave.mk:481: target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb] Error 130 make[1]: Leaving directory '/home/zbq/sonic/sonic-buildimage'

D DST_IP=172.16.1.1 cmd=sudo iptables -t nat -n -L OUTPUT 1 out=DNAT tcp -- 0.0.0.0/0 172.16.1.1 tcp dpt:3128 to:11.11.11.11:8080 err= cmd=sudo iptables -t nat -D OUTPUT 1 out= err= cmd=sudo iptables -t nat -n -L OUTPUT 1 out= err= cmd=sudo iptables -t nat -A OUTPUT -p tcp -d 172.16.1.1 --dport 3128 -j DNAT --to-destination 103.252.115.221:3128 out= err=

----------- coverage: platform linux, python 3.9.2-final-0 ----------- Name Stmts Miss Cover

ctrmgr/init.py 0 0 100% ctrmgr/container 206 8 96% ctrmgr/container_startup.py 142 4 97% ctrmgr/ctrmgr_iptables.py 87 12 86% ctrmgr/ctrmgr_tools.py 99 1 99% ctrmgr/ctrmgrd.py 313 15 95% ctrmgr/kube_commands.py 225 14 94%

TOTAL 1072 54 95% Coverage HTML written to dir htmlcov Coverage XML written to file coverage.xml

=========================== short test summary info ============================ FAILED tests/ctrmgr_iptables_test.py::TestIPTableUpdate::test_table - Asserti... ======================== 1 failed, 15 passed in 10.97s ========================= [ FAIL LOG END ] [ target/python-wheels/bullseye/sonic_ctrmgrd-1.0.0-py3-none-any.whl ] make: [slave.mk:667: target/python-wheels/bullseye/sonic_ctrmgrd-1.0.0-py3-none-any.whl] Error 1 make: Waiting for unfinished jobs.... [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] [ 02 ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] [ 03 ] [ target/debs/bullseye/frr_7.5.1-sonic-0_amd64.deb ] [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] [ 02 ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] [ FAIL LOG START ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] [ REASON ] : target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb does not exist
[ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] make[1]: warning: -j2 forced in submake: resetting jobserver mode. make[1]: Entering directory '/sonic/src/radius/pam' Build freeradius-server v1.1.x pushd freeradius rm -rf freeradius-server git clone https://github.com/FreeRADIUS/freeradius-server.git popd pushd freeradius/freeradius-server git checkout -f 5f715dba4d2dbdb268bf60fcc656352274930941 Apply patch patch -p1 < ../patches/freeradius_configure.patch patch -p1 < ../patches/freeradius_2007-04-06.patch patch -p1 < ../patches/freeradius_libeap_deprecated_openssl_1_0.patch cp ../patches/config.sub . cp ../patches/config.guess . ./configure --disable-static --enable-libtool-lock make popd Obtain pam_radius rm -rf ./pam_radius git clone https://github.com/FreeRADIUS/pam_radius.git pushd ./pam_radius This is the latest commit with Radius MPL support git checkout -f 149c25df84cf5cd0e9addd9346699a9ca8fdddd2 cp -r ../debian . dpkg-buildpackage -rfakeroot -b -us -uc -nc popd mv libpam-radius-auth_1.4.1-1_amd64.deb /sonic/target/debs/bullseye/ /sonic/src/radius/pam/freeradius /sonic/src/radius/pam Cloning into 'freeradius-server'... error: 3888 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed make[1]: [Makefile:8: /sonic/target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb] Error 128 make[1]: Leaving directory '/sonic/src/radius/pam' [ FAIL LOG END ] [ target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb ] make: [slave.mk:481: target/debs/bullseye/libpam-radius-auth_1.4.1-1_amd64.deb] Error 1 [ 01 ] [ target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb ] ^Cmake: *** [slave.mk:481: target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb] Error 130 make[1]: Leaving directory '/home/zbq/sonic/sonic-buildimage'

What's the wrong? Is it so difficult to build the sonic all? How to build the sonic by steps?

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

zhengbiqing commented 2 years ago

the recent run "make SONIC_BUILD_JOBS=4 all", after about 20 minutes, exist with error. the tail of the output is :

[ 02 ] [ target/debs/bullseye/libnl-3-200_3.5.0-1_amd64.deb-install ] [ 03 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] [ 04 ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] [ FAIL LOG START ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] [ REASON ] : target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb does not exist NON-EXISTENT PREREQUISITES: target/debs/bullseye/linux-headers-5.10.0-8-2-amd64_5.10.46-4_amd64.deb-install target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb-install [ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] /sonic/platform/broadcom/sonic-platform-modules-arista /sonic dpkg-buildpackage: error: cannot open file debian/changelog: No such file or directory [ FAIL LOG END ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] make: [slave.mk:520: target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb] Error 1 make: Waiting for unfinished jobs.... [ 01 ] [ target/debs/bullseye/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ] [ 02 ] [ target/debs/bullseye/libnl-3-200_3.5.0-1_amd64.deb-install ] [ 03 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] [ 01 ] [ target/debs/bullseye/libnl-3-200_3.5.0-1_amd64.deb-install ] [ 02 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] [ 01 ] [ target/python-wheels/bullseye/swsssdk-2.0.1-py3-none-any.whl-install ] make[1]: Leaving directory '/home/zbq/sonic/sonic-buildimage'

the all output is: make-output.txt

zhengbiqing commented 2 years ago

I try "make SONIC_BUILD_JOBS=1 all". after do something. the command "stop": no output and no error, and the command not exist. "stop" for several hours:

Successfully built sonic-host-services pycairo Installing collected packages: pycairo, sonic-host-services Successfully installed pycairo-1.21.0 sonic-host-services-1.0 Found existing installation: sonic-host-services 1.0 Uninstalling sonic-host-services-1.0: [ 01 ] [ target/sonic-broadcom.bin ]

It seems the command will stay there for ever.

this is all of the output: make-all-log-1.txt

zhengbiqing commented 2 years ago

At last, the command exist with error:

Uninstalling sonic-host-services-1.0: [ 01 ] [ target/sonic-broadcom.bin ] [ FAIL LOG START ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] [ REASON ] : target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb does not exist NON-EXISTENT PREREQUISITES: target/debs/bullseye/linux-headers-5.10.0-8-2-amd64_5.10.46-4_amd64.deb-install target/debs/bullseye/linux-headers-5.10.0-8-2-common_5.10.46-4_all.deb-install [ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] /sonic/platform/broadcom/sonic-platform-modules-arista /sonic dpkg-buildpackage: error: cannot open file debian/changelog: No such file or directory [ FAIL LOG END ] [ target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb ] make: [slave.mk:520: target/debs/bullseye/sonic-platform-arista_1.0_amd64.deb] Error 1 make[1]: [Makefile.work:312: all] Error 2 make[1]: Leaving directory '/home/zbq/sonic/sonic-buildimage' make: *** [Makefile:33: all] Error 2

Anyone can help me?

prsunny commented 2 years ago

[Issue triage] Can you try with building a specific target image instead of "make all" ?