op-build fails to build pnor image when enabling console tracing in hostboot code

pridhiviraj commented 7 years ago

Tried building pnor image by enabling console tracing in hostboot code, it fails with below errors. """ERROR: PnorUtils::checkSpaceConstraints: Image provided (/home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot_extended.header.bin.ecc) has size (11796480) which is greater than allocated space (5898240) for section=HBI. Aborting! at /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/PnorUtils.pm line 352."""

op-build commit id:43882ebce3063d3b1c9353f598a61f4fa557dd71

/home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images//buildpnor.pl --pnorOutBin /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/pnor/firestone.pnor --pnorLayout /home/ubuntu/tracing/op-build/output/build/openpower-pnor-cdfe37976dae7d3171ce9b999cf91f1b5d80e9cc/defaultPnorLayoutWithGoldenSide.xml --binFile_HBD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//FIRESTONE_HB.targeting.bin.ecc --binFile_SBE /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//venice_sbe.img.ecc --binFile_SBEC /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//centaur_sbec_pad.img.ecc --binFile_WINK /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//p8.ref_image.hdr.bin.ecc --binFile_HBB /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot.header.bin.ecc --binFile_HBI /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot_extended.header.bin.ecc --binFile_HBRT /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot_runtime.header.bin.ecc --binFile_HBEL /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hbel.bin.ecc --binFile_GUARD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//guard.bin.ecc --binFile_PAYLOAD /home/ubuntu/tracing/op-build/output/images/skiboot.lid.xz --binFile_BOOTKERNEL /home/ubuntu/tracing/op-build/output/images/zImage.epapr --binFile_NVRAM /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//nvram.bin --binFile_MVPD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//mvpd_fill.bin.ecc --binFile_DJVPD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//djvpd_fill.bin.ecc --binFile_CVPD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//cvpd.bin.ecc --binFile_ATTR_TMP /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//attr_tmp.bin.ecc --binFile_ATTR_PERM /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//attr_perm.bin.ecc --binFile_OCC /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/occ/occ.bin.ecc --binFile_FIRDATA /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//firdata.bin.ecc --binFile_CAPP /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//cappucode.bin.ecc --binFile_SECBOOT /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//secboot.bin.ecc --binFile_VERSION /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_version/openpower-pnor.version.txt --fpartCmd "fpart" --fcpCmd "fcp"

TRACE: PnorUtils::loadPnorLayout: metadata: imageSize = 67108864, blockSize=4096, arrangement = A-D-B, numOfSides: 2, sideSize: 33554432, tocSize: 32768 TRACE: A-D-B: side:A HBB:32899072, primaryTOC:0, backupTOC:33521664, golden: no TRACE: A-D-B: side:B HBB:66482176, primaryTOC:33554432, backupTOC:67076096, golden: yes ERROR: PnorUtils::checkSpaceConstraints: Image provided (/home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot_extended.header.bin.ecc) has size (11796480) which is greater than allocated space (5898240) for section=HBI. Aborting! at /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/PnorUtils.pm line 352. Error running command: /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images//buildpnor.pl --pnorOutBin /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/pnor/firestone.pnor --pnorLayout /home/ubuntu/tracing/op-build/output/build/openpower-pnor-cdfe37976dae7d3171ce9b999cf91f1b5d80e9cc/defaultPnorLayoutWithGoldenSide.xml --binFile_HBD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//FIRESTONE_HB.targeting.bin.ecc --binFile_SBE /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//venice_sbe.img.ecc --binFile_SBEC /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//centaur_sbec_pad.img.ecc --binFile_WINK /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//p8.ref_image.hdr.bin.ecc --binFile_HBB /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot.header.bin.ecc --binFile_HBI /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot_extended.header.bin.ecc --binFile_HBRT /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hostboot_runtime.header.bin.ecc --binFile_HBEL /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//hbel.bin.ecc --binFile_GUARD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//guard.bin.ecc --binFile_PAYLOAD /home/ubuntu/tracing/op-build/output/images/skiboot.lid.xz --binFile_BOOTKERNEL /home/ubuntu/tracing/op-build/output/images/zImage.epapr --binFile_NVRAM /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//nvram.bin --binFile_MVPD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//mvpd_fill.bin.ecc --binFile_DJVPD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//djvpd_fill.bin.ecc --binFile_CVPD /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//cvpd.bin.ecc --binFile_ATTR_TMP /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//attr_tmp.bin.ecc --binFile_ATTR_PERM /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//attr_perm.bin.ecc --binFile_OCC /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/occ/occ.bin.ecc --binFile_FIRDATA /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//firdata.bin.ecc --binFile_CAPP /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//cappucode.bin.ecc --binFile_SECBOOT /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_pnor_scratch//secboot.bin.ecc --binFile_VERSION /home/ubuntu/tracing/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_version/openpower-pnor.version.txt --fpartCmd "fpart" --fcpCmd "fcp". Nonzero return code of (65280) returned. make[1]: * [/home/ubuntu/tracing/op-build/output/build/openpower-pnor-cdfe37976dae7d3171ce9b999cf91f1b5d80e9cc/.stamp_images_installed] Error 255 make: * [_all] Error 2 make: Leaving directory `/home/ubuntu/tracing/op-build/buildroot'

dcrowell77 commented 7 years ago

Unfortunately we've reached the tipping point where we no longer have enough space defined in the pnor to hold all of the trace messages. There are ways to allocate more space, but they are not compatible with code updates so we can't make those changes permanently.

If you are building your own pnor images and using pflash to update it (versus hpm or another more formal method) then you can modify output/build/openpower-pnor-b937782225b96b6fb176f8acb59278026c8d9fd6/defaultPnorLayoutWithGoldenSide.xml to increase the size of the HBI partition by stealing space from other partitions or moving it around.

ploetzma commented 7 years ago

One solution we tried a while back was to move around the pnor partitions to make space for traces. This did work at the time for hostboot, but it broke HPM updates.

You can try pulling this fork down and building. It hasn't been updated in a while so not sure if it is current or not. https://github.com/ploetzma/pnor/commit/ee6d1a61aa33f8ce760b7994c1b3dff0f435473b

In the mean time - I believe it was in AMI's queue to fix their update path. @mzipse was that done?

pragupta commented 5 years ago

We stopped compiling with traces enabled because of this, instead, we use a tool called tracelite to get the traces. I will work on pushing the tracelite changes to github for external consumption.

popfuture commented 5 years ago

I will make the tracelite weave tool available, so that people can start using that for traces instead.

stintel commented 4 years ago

I will make the tracelite weave tool available, so that people can start using that for traces instead.

Where can we find this tool?

dcrowell77 commented 4 years ago

Unfortunately the trace-lite never got 100% finished and the developer that was working on it is off on something else now. Most of the pieces are there but it isn't turnkey consumable. I'll attach the source code I could find for the weave tool. Look for the term "trace_lite" (upper and lower case) in the hostboot code to see how and where things run.

To get the traces from the SOL output:

Capture the SOL output to a file with tee or your favorite terminal program
Pass the host.sol output file to weave: e.g
- cat host.sol | weave .../op-build/op-build/output/build/hostboot-xxx/img/hbotStringFile > hbTrace

We'd welcome any contributions that would finish this off. The vision would be that you pipe the console output through a program that can decode it on the fly so you get the best of both worlds - smaller pnor footprint and full logs.

weave.cpp.gz

open-power / hostboot

op-build fails to build pnor image when enabling console tracing in hostboot code #63