Problem during compile photon

wrednybarteK commented 2 years ago

Describe the bug

When compiling photono from source files, the process is always interrupted at a random point after a random time beyond a few hours. Various packages are interrupted and restarting the build eventually terminates the process with a positive result.

Ubuntu 18.04.5 LTS

Reproduction steps

1.All steps from https://vmware.github.io/photon/assets/files/html/3.0/photon_installation/build_the_iso.html
2.sudo make iso
3.get an error

Expected behavior

Finish building proces with out any issue.

Additional context

For example one error during last building proces `Building package : kubernetes Building rpm is failed /usr/src/photon/SPECS/kubernetes.spec Failed while building rpm:kubernetes Failed while building package: kubernetes warning: Macro expanded in comment on line 192: %{_bindir}/kubefed

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.n7N9wP

umask 022
cd /usr/src/photon/BUILD
cd /usr/src/photon/BUILD
rm -rf kubernetes-1.19.15
/bin/gzip -dc /usr/src/photon/SOURCES/kubernetes-1.19.15.tar.gz
/bin/tar --no-same-owner -xof -
STATUS=0
'[' 0 -ne 0 ']'
cd kubernetes-1.19.15
/bin/chmod -Rf a+rX,u+w,g-w,o-w .
cd ..
tar xf /usr/src/photon/SOURCES/contrib-0.7.0.tar.gz --no-same-owner
sed -i -e 's|127.0.0.1:4001|127.0.0.1:2379|g' contrib-0.7.0/init/systemd/environ/apiserver
sed -i /KUBE_ALLOW_PRIV/d contrib-0.7.0/init/systemd/kubelet.service
RPM_EC=0 ++ jobs -p
exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.FaC4aT
umask 022
cd /usr/src/photon/BUILD
cd kubernetes-1.19.15
make -j12 +++ [0401 19:57:54] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/deepcopy-gen +++ [0401 19:57:54] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen +++ [0401 19:57:54] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/conversion-gen +++ [0401 19:57:54] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/defaulter-gen +++ [0401 19:57:54] Building go targets for linux/amd64: ./vendor/k8s.io/kube-openapi/cmd/openapi-gen +++ [0401 19:57:54] Building go targets for linux/amd64: ./vendor/github.com/go-bindata/go-bindata/go-bindata F0401 19:58:07.436569 124086 defaulter.go:292] cannot import package ../../../../vendor/k8s.io/api/core/v1 goroutine 1 [running]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000010001, 0xc00d315800, 0x6c, 0xbb) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:996 +0xb9 k8s.io/kubernetes/vendor/k8s.io/klog/v2.(loggingT).output(0x96c760, 0xc000000003, 0x0, 0x0, 0xc02299e460, 0x83127e, 0xc, 0x124, 0x0) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:945 +0x191 k8s.io/kubernetes/vendor/k8s.io/klog/v2.(loggingT).printf(0x96c760, 0x3, 0x0, 0x0, 0x7699c0, 0x18, 0xc02269fbe8, 0x1, 0x1) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x17a k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1456 k8s.io/kubernetes/vendor/k8s.io/gengo/examples/defaulter-gen/generators.Packages(0xc00fa673b0, 0xc000012fa0, 0x74aa2d, 0x6, 0xc00fa673b0) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/gengo/examples/defaulter-gen/generators/defaulter.go:292 +0x1a05 k8s.io/kubernetes/vendor/k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc000012fa0, 0xc0000a6ff0, 0x74aa2d, 0x6, 0x777a68, 0x0, 0x0) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/gengo/args/args.go:206 +0x1b7 main.main() _output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter-gen/main.go:76 +0x32b

goroutine 6 [chan receive]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x96c760) /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0 /usr/src/photon/BUILD/kubernetes-1.19.15/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:416 +0xd8 make[1]: [Makefile.generated_files:332: gen_defaulter] Error 1 make[1]: Waiting for unfinished jobs.... make: *** [Makefile:546: generated_files] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.FaC4aT (%build)

RPM build errors: Macro expanded in comment on line 192: %{_bindir}/kubefed

Bad exit status from /var/tmp/rpm-tmp.FaC4aT (%build)

RPM build failed Traceback (most recent call last): File "/root/photon/support/package-builder/PackageBuilder.py", line 40, in build self._buildPackage() File "/root/photon/support/package-builder/PackageBuilder.py", line 77, in _buildPackage raise e File "/root/photon/support/package-builder/PackageBuilder.py", line 65, in _buildPackage self.logPath) File "/root/photon/support/package-builder/PackageUtils.py", line 140, in buildRPMSForGivenPackage raise e File "/root/photon/support/package-builder/PackageUtils.py", line 132, in buildRPMSForGivenPackage logFilePath, package, version, macros) File "/root/photon/support/package-builder/PackageUtils.py", line 319, in _buildRPM raise Exception("RPM build failed") Exception: RPM build failed RPM build failed Traceback (most recent call last): File "/root/photon/support/package-builder/WorkerThread.py", line 29, in run pkgBuilder.build(pkg, doneList) File "/root/photon/support/package-builder/PackageBuilder.py", line 44, in build raise e File "/root/photon/support/package-builder/PackageBuilder.py", line 40, in build self._buildPackage() File "/root/photon/support/package-builder/PackageBuilder.py", line 77, in _buildPackage raise e File "/root/photon/support/package-builder/PackageBuilder.py", line 65, in _buildPackage self.logPath) File "/root/photon/support/package-builder/PackageUtils.py", line 140, in buildRPMSForGivenPackage raise e File "/root/photon/support/package-builder/PackageUtils.py", line 132, in buildRPMSForGivenPackage logFilePath, package, version, macros) File "/root/photon/support/package-builder/PackageUtils.py", line 319, in _buildRPM raise Exception("RPM build failed") Exception: RPM build failed Some of the packages failed: ['kubernetes-1.19.15'] Failed during building package Traceback (most recent call last): File "./build.py", line 1221, in main buildImage.build_iso() File "./build.py", line 935, in build_iso rpmBuildTarget.packages() File "./build.py", line 626, in packages self.logger) File "/root/photon/support/package-builder/builder.py", line 47, in buildPackagesForAllSpecs Builder.buildSpecifiedPackages(listPackages, buildThreads, pkgBuildType, pkgInfoJsonFile, logger) File "/root/photon/support/package-builder/builder.py", line 23, in buildSpecifiedPackages pkgManager.buildPackages(listPackages, buildThreads) File "/root/photon/support/package-builder/PackageManager.py", line 107, in buildPackages self._buildGivenPackages(listPackages, buildThreads) File "/root/photon/support/package-builder/PackageManager.py", line 202, in _buildGivenPackages self._buildPackages(buildThreads) File "/root/photon/support/package-builder/PackageManager.py", line 237, in _buildPackages raise Exception("Failed during building package") Exception: Failed during building package Makefile:6: recipe for target 'iso' failed make: *** [iso] Error 1`

Disk space is fine, memory too, I couldn't find any issue with host

dcasota commented 2 years ago

according to the change log there is a duplicate package name 1.18.19-13, might correlate to go issue 51836?

wrednybarteK commented 2 years ago

Tried a new version but still Building.log

dcasota commented 2 years ago

tried to reach out to @prashant1221. @gpiyush-dev or him may help here to fix the duplicate package name 1.18.19-13 ? The version before and the version should remain eg. something like 1.18.19.12 (existing), 1.18.19.13 (existing), 1.18.19.14(fixed typo), 1.23.2-1 (existing), 1.23.2-2 (fixed typo). Unfortunately I'm not familiar with the devops tecnique to assemble a proper pull request. Help for the go1.18 backport fix would be somewhat helpful, too.

prashant1221 commented 2 years ago

tried to reach out to @prashant1221. @gpiyush-dev or him may help here to fix the duplicate package name 1.18.19-13 ? The version before and the version should remain eg. something like 1.18.19.12 (existing), 1.18.19.13 (existing), 1.18.19.14(fixed typo), 1.23.2-1 (existing), 1.23.2-2 (fixed typo). Unfortunately I'm not familiar with the devops tecnique to assemble a proper pull request. Help for the go1.18 backport fix would be somewhat helpful, too.

Hi Daniel, thanks for the finding I'll push the fix right away.

prashant1221 commented 2 years ago

Tried a new version but still Building.log

@wrednybarteK This is an OOM issue. I am updating k8s in dev/master branch.

But as a POC can you try this fix 'https://github.com/vmware/photon/commit/500ff39737e139620cded2fe7522e64e1f3a532f' and build.
Basically we are decreasing the make-flags count, so no. of concurrent process during build time will decrease. If it still fails we can build without providing any flags just 'make {target}'. Then the build should be successful in your machine.

vmware / photon