search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
715 stars 1.37k forks source link

Failing in building 202111 #16259

Open guichuan2018 opened 1 year ago

guichuan2018 commented 1 year ago

Description

Tried to build a sonic image off branch 202111 with platform=innovium and NOSTRETCH=1, multiple build errors were seen. First a debian tar file was unable to find as it's absent from the URL. Second, after applying patch to the above issue, the second error occurred in rebuilding the image. The second error is that a python test file tried to use a function that's not defined in the 202111 repo.

Steps to reproduce the issue:

  1. umask 22
  2. git clone --recurse-submodules https://github.com/sonic-net/sonic-buildimage.git
  3. cd sonic-buildimage
  4. git checkout 202111 --force
  5. make init 
make configure PLATFORM=innovium NOSTRETCH=1 
make target/sonic-innovium.bin NOSTRETCH=1

Describe the results you received:

1) the following error was see in "running make target/sonic-innovium.bin NOSTRETCH=1' :

2023-08-10 07:07:37  http://http.debian.net/debian/pool/main/h/hiredis/hiredis_0.14.0-3~bpo9+1.debian.tar.xz
Resolving http.debian.net (http.debian.net)... 130.89.148.77, 128.31.0.62, 2001:67c:2564:a119::77, ...
Connecting to http.debian.net (http.debian.net)|130.89.148.77|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://cdn-fastly.deb.debian.org/debian/pool/main/h/hiredis/hiredis_0.14.0-3~bpo9+1.debian.tar.xz [following]
 2023-08-10 07:07:37   http://cdn-fastly.deb.debian.org/debian/pool/main/h/hiredis/hiredis_0.14.0-3~bpo9+1.debian.tar.xz
Resolving cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)... 199.232.94.132, 2a04:4e42:57::644
Connecting to cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)|199.232.94.132|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-08-10 07:07:37 ERROR 404: Not Found.

2) Applied the following patch to address the above error:

diff --git a/rules/hiredis.mk b/rules/hiredis.mk
index 71a18a1d6aac..a97e0bf1bb83 100644
--- a/rules/hiredis.mk
+++ b/rules/hiredis.mk
@@ -1,7 +1,7 @@
 # libhiredis package

-HIREDIS_VERSION = 0.14.0
-HIREDIS_VERSION_FULL = $(HIREDIS_VERSION)-3~bpo9+1
+HIREDIS_VERSION = 0.14.1
+HIREDIS_VERSION_FULL = ${HIREDIS_VERSION}-1

 export HIREDIS_VERSION HIREDIS_VERSION_FULL

@@ -12,6 +12,7 @@ SONIC_MAKE_DEBS += $(LIBHIREDIS)
 LIBHIREDIS_DEV = libhiredis-dev_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARCH).deb
 $(eval $(call add_derived_package,$(LIBHIREDIS),$(LIBHIREDIS_DEV)))

-LIBHIREDIS_DBG = libhiredis-dbg_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARCH).deb
 LIBHIREDIS_DBG = libhiredis0.14-dbgsym_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARCH).deb
 $(eval $(call add_derived_package,$(LIBHIREDIS),$(LIBHIREDIS_DBG)))
+
+export LIBHIREDIS LIBHIREDIS_DEV LIBHIREDIS_DBG
diff --git a/src/hiredis/Makefile b/src/hiredis/Makefile
index 4935a039e0af..ad6900554132 100644
--- a/src/hiredis/Makefile
+++ b/src/hiredis/Makefile
@@ -2,21 +2,23 @@
 SHELL = /bin/bash
 .SHELLFLAGS += -e

-MAIN_TARGET = libhiredis0.14_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARCH).deb
-DERIVED_TARGETS = libhiredis0.14-dbgsym_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARCH).deb libhiredis-dev_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARCH).deb
+MAIN_TARGET = $(LIBHIREDIS)
+DERIVED_TARGETS = $(LIBHIREDIS_DBG) $(LIBHIREDIS_DEV)

 $(addprefix $(DEST)/, $(MAIN_TARGET)): $(DEST)/% :
-   rm -rf hiredis-$(HIREDIS_VERSION)
+   # Remove any stale files
+   rm -rf ./hiredis-$(HIREDIS_VERSION)

-   wget -O hiredis_$(HIREDIS_VERSION).orig.tar.gz http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION).orig.tar.gz
-   wget -O hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz
-   wget -O hiredis_$(HIREDIS_VERSION_FULL).dsc http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).dsc
+   # Get hiredis release, debian files
+   dget -u http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).dsc
+
+   # Build source and Debian packages
+   pushd ./hiredis-$(HIREDIS_VERSION)

-   dpkg-source -x hiredis_$(HIREDIS_VERSION_FULL).dsc
-   pushd hiredis-$(HIREDIS_VERSION)
    dpkg-buildpackage -rfakeroot -d -b -us -uc -j$(SONIC_CONFIG_MAKE_JOBS) --admindir $(SONIC_DPKG_ADMINDIR)
    popd

+   # Move the newly-built .deb packages to the destination directory
    mv $* $(DERIVED_TARGETS) $(DEST)/

 $(addprefix $(DEST)/, $(DERIVED_TARGETS)): $(DEST)/% : $(DEST)/$(MAIN_TARGET)

3) With the above patch, the above build error was gone, however new errors were seen in building the target image:

____________ ERROR collecting tests/hostcfgd/hostcfgd_fips_test.py _____________^M
ImportError while importing test module '/sonic/src/sonic-host-services/tests/hostcfgd/hostcfgd_fips_test.py'.^M
Hint: make sure your test modules/packages have valid Python names.^M
Traceback:^M
/usr/lib/python3.9/importlib/__init__.py:127: in import_module^M
    return _bootstrap._gcd_import(name[level:], package, level)^M
tests/hostcfgd/hostcfgd_fips_test.py:14: in <module>^M
    from sonic_py_common.general import getstatusoutput_noshell^M
E   ImportError: cannot import name 'getstatusoutput_noshell' from 'sonic_py_common.general' (/usr/local/lib/python3.9/dist-packages/sonic_py_common/general.py)^M

It complains that it can not import getstatusoutput_noshell. The problem is it is not defined at all in the working repo of 202111. Thus is will result in a import error. The question is why it is used by importing while it's not actually defined?

Describe the results you expected:

Expected the image to be successfully built without any patch.

The following is the official build which was successful: https://dev.azure.com/mssonic/build/_build/results?buildId=344942&view=logs&j=1661729d-42ae-5bdb-a427-0e14292534eb

Or the above official build used any patches to make it build?

Output of show version:

no show version as it's about build errors

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

guichuan2018 commented 1 year ago

It's been days. Looks like nobody took a look at it.

Does it mean this is not an issue?

If not an issue, how come it failed?

vmittal-msft commented 1 year ago

@kishoreatreya Can you please help take a look ?

kishoreatreya commented 1 year ago

I'm asking our team to look. Stay tuned.

Keshavg-marvell commented 1 year ago

@guichuan2018 Thanks for reporting.

I will try to compile locally and look in to it. But I am just curious why 2022111 is passing in the nightly community runs mentioned here https://github.com/sonic-net/sonic-buildimage

Just wondering what is the difference between what you are trying and nightly run. Does nightly runs picks packages from some local caches and not from debian.org or some other official repo?

I suspect some package are removed/updated which is reference by sonic repo. But I will look in to this and find out more.

guichuan2018 commented 1 year ago

@Keshavg-marvell

Thank you for looking into this build failing. So you have seen the same build failures, right?

There is no difference between what I'm trying and the nightly as they are all based on 202111.

Not sure how nightly runs picks packages.

Is there any way we can check with the folks who're running nightly build on it?

Keshavg-marvell commented 1 year ago

@guichuan2018 Not really I actually faced only missing package issue for hiredis in deb.org as you faced. By pointing to archive.debian.org repo for hiredis i could build innovium sonic successfully. I believe nightly runs are done by Microsoft in azure infra and internally may getting debian/packaged from cache repo that's why nightly is passing but manual runs fails. Let us know if it is indeed the case.

With your git pull steps I was facing errors/warning and I suspect it may not be checking out sonic-host-services submodule correctly. Below is the steps i followed and it build innovium build successfully. Please try the same and let us know if it works for you as well.

Steps: git clone https://github.com/Azure/sonic-buildimage.git -b 202111 cd sonic-buildimage/ make init make configure PLATFORM=innovium NOSTRETCH=1 make target/sonic-innovium.bin NOSTRETCH=1

Below is the diffset you need to apply where package is retrieved from archieve.debian.org instead of getting it from http.debian.net after doing make init.

diff --git a/src/hiredis/Makefile b/src/hiredis/Makefile
 index 4935a039e..20c5ea281 100644
 --- a/src/hiredis/Makefile
 +++ b/src/hiredis/Makefile
 @@ -8,9 +8,9 @@ DERIVED_TARGETS = libhiredis0.14-dbgsym_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARC
  $(addprefix $(DEST)/, $(MAIN_TARGET)): $(DEST)/% :
         rm -rf hiredis-$(HIREDIS_VERSION)

 -       wget -O hiredis_$(HIREDIS_VERSION).orig.tar.gz http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION).orig.tar.gz
 -       wget -O hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz
 -       wget -O hiredis_$(HIREDIS_VERSION_FULL).dsc http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).dsc
 +       wget -O hiredis_$(HIREDIS_VERSION).orig.tar.gz http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION).orig.tar.gz
 +       wget -O hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz
 +       wget -O hiredis_$(HIREDIS_VERSION_FULL).dsc http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).dsc

        dpkg-source -x hiredis_$(HIREDIS_VERSION_FULL).dsc
        pushd hiredis-$(HIREDIS_VERSION)
guichuan2018 commented 1 year ago

@Keshavg-marvell

Thanks for providing the patch. I followed the steps you suggested and applied the patch, but still saw an error below. You didn't see this error?


"SONIC_DPKG_CACHE_METHOD"         : "none"

[ FAIL LOG START ] [ target/debs/buster/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ]
target/debs/buster/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb does not exist
[  FAIL LOG END  ] [ target/debs/buster/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install ]
make: *** [slave.mk:605: target/debs/buster/libhiredis0.14_0.14.0-3~bpo9+1_amd64.deb-install] Error 1
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
make[1]: *** [Makefile.work:333: buster] Error 2
make[1]: Leaving directory '/home/u1/x/sonic/innovium4/sonic-buildimage'
make: *** [Makefile:32: target/sonic-innovium.bin] Error 2
Keshavg-marvell commented 1 year ago

No i don;t hit this issue. for me it works and get the debian package correctly.

Keshavg-marvell commented 1 year ago

Most like this could be due to Makefile issue which is sensitive to space and tabs. please check if the below changes applied correctly. 

diff --git a/src/hiredis/Makefile b/src/hiredis/Makefile
index 4935a039e..20c5ea281 100644
--- a/src/hiredis/Makefile
+++ b/src/hiredis/Makefile
@@ -8,9 +8,9 @@ DERIVED_TARGETS = libhiredis0.14-dbgsym_$(HIREDIS_VERSION_FULL)_$(CONFIGURED_ARC
 $(addprefix $(DEST)/, $(MAIN_TARGET)): $(DEST)/% :
        rm -rf hiredis-$(HIREDIS_VERSION)

-       wget -O hiredis_$(HIREDIS_VERSION).orig.tar.gz http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION).orig.tar.gz
-       wget -O hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz
-       wget -O hiredis_$(HIREDIS_VERSION_FULL).dsc http://http.debian.net/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).dsc
+       wget -O hiredis_$(HIREDIS_VERSION).orig.tar.gz http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION).orig.tar.gz
+       wget -O hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).debian.tar.xz
+       wget -O hiredis_$(HIREDIS_VERSION_FULL).dsc http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_$(HIREDIS_VERSION_FULL).dsc

        dpkg-source -x hiredis_$(HIREDIS_VERSION_FULL).dsc
        pushd hiredis-$(HIREDIS_VERSION)

just to rule out any blocking of URL you can check manually if you could download these hiredis debians like below

wget -O hiredis_0.14.0-3~bpo9+1.dsc http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_0.14.0-3~bpo9+1.dsc
wget -O hiredis_0.14.0-3~bpo9+1.debian.tar.xz http://archive.debian.org/debian/pool/main/h/hiredis/hiredis_0.14.0-3~bpo9+1.debian.tar.xz

Let us know if you still face the issue and shared the diff of changes.