pridhiviraj commented 7 years ago

copy the firestone xml where the common lives

bash -c 'mkdir -p /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch && cp -r /home/pridhiviraj/op-build/output/build/firestone-xml-2494a430656504d014780000adbc2c2e7cfc54ab/* /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch'

generate the system mrw xml

perl -I /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl -x /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/firestone.xml Loading MRW XML: /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/firestone.xml Use of uninitialized value in new at /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl line 664. Use of uninitialized value in new at /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl line 665. Use of uninitialized value in new at /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl line 666. Use of uninitialized value in new at /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl line 664. Use of uninitialized value in new at /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl line 665. Use of uninitialized value in new at /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/processMrw.pl line 666. Creating XML: /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/FIRESTONE_hb.mrw.xml MRW created successfully!

merge in any system specific attributes, hostboot attributes

/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/mergexml.sh /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/"FIRESTONE_hb.system.xml" /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/attribute_types.xml /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/attribute_types_hb.xml /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/target_types_merged.xml /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/target_types_hb.xml /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/"FIRESTONE_hb.mrw.xml" > /home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml;

creating the targeting binary

/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/xmltohb.pl --hb-xml-file=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml --fapi-attributes-xml-file=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/fapiattrs.xml --src-output-dir=none --img-output-dir=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/ --vmm-consts-file=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/vmmconst.h --noshort-enums --bios-xml-file=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/"FIRESTONE_bios.xml" --bios-schema-file=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/bios.xsd --bios-output-file=/home/pridhiviraj/op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/"firestone"_bios_metadata.xml [FATAL!] Could not find enumeration with ID of IBSCOM_MCS_BASE_ADDR

 1: main::getEnumerationType(1298)
 2: main::packSingleSimpleTypeAttribute(4749)
 3: main::packAttribute(4829)
 4: main::generateTargetingImage(5532)
 5: (379)

package/pkg-generic.mk:216: recipe for target '/home/pridhiviraj/op-build/output/build/firestone-xml-2494a430656504d014780000adbc2c2e7cfc54ab/.stamp_built' failed make[1]: [/home/pridhiviraj/op-build/output/build/firestone-xml-2494a430656504d014780000adbc2c2e7cfc54ab/.stamp_built] Error 1 Makefile:76: recipe for target '_all' failed make: [_all] Error 2 make: Leaving directory '/home/pridhiviraj/op-build/buildroot'

dcrowell77 commented 7 years ago

I'm completely confused on what you ran to hit this. This is from a regular op-build command? What level? Any user-level changes?

pridhiviraj commented 7 years ago

No changes. I just build using regular op-build command, Cloned latest op-build master-next branch and tried building PNOR image for firestone platforms

git clone --recursive https://github.com/open-power/op-build --branch master-next . op-build-env op-build firestone_defconfig && op-build

Able to reproduce second time too.

dcrowell77 commented 7 years ago

I ran these exact command and it all worked fine. What OS/machinetype are you running on?

Attach these files from your build tree: op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml op-build/output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/FIRESTONE_hb.mrw.xml op-build/output/build/firestone-xml-2494a430656504d014780000adbc2c2e7cfc54ab/firestone.xml

pridhiviraj commented 7 years ago

uname -a Linux localhost.localdomain 4.6.7-300.fc24.x86_64 #1 SMP Wed Aug 17 18:48:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I am running on my Laptop having fedora 24 on x86_64 architecture.

temporary_hb.hb.txt FIRESTONE_hb.mrw.txt firestone.txt

williamspatrick commented 7 years ago

@stewart-ibm - Is this the distro that had a broken PERL XML parser that we had to work around?

ghost commented 7 years ago

Patrick Williams notifications@github.com writes:

@stewart-ibm - Is this the distro that had a broken PERL XML parser that we had to work around?

It's possible. I think it was also broken in Fedora 24. An update has gone out, but I'm not sure if it's gone to Fedora 24 also (or just 25) or if it has been applied.

-- Stewart Smith OPAL Architect, IBM.

Over-enthusiastic commented 7 years ago

I also see this error on Ubuntu 17.04 . DISTRIB_DESCRIPTION="Ubuntu 17.04" NAME="Ubuntu" VERSION="17.04 (Zesty Zapus)" ID=ubuntu Following are the xml parser versions which are installed : libxml-libxml-perl/zesty,now 2.0128+dfsg-1build1 amd64 [installed,automatic] libxml-namespacesupport-perl/zesty,zesty,now 1.11-1 all [installed,automatic] libxml-parser-perl/zesty,now 2.44-2build2 amd64 [installed,automatic] libxml-sax-base-perl/zesty,zesty,now 1.07-1 all [installed,automatic] What is the correct xml parser version which should be used?

ajdlinux commented 7 years ago

Apparently this might be occurring on Debian too - mentioned on IRC that someone ran into this.

cyrilbur-ibm commented 6 years ago

I've hit this too trying to build for palmetto.

Can we get an answer to which XML parser we need?

All I did was:

$ git clone --recursive https://github.com/open-power/op-build
$ . op-build-env
$ op-build palmetto_defconfig && op-build

What I pulled:

$ git show
commit cbac09a858a7c8773605ab1a36bd1ad1a576fa65 (HEAD -> master, origin/master, origin/HEAD)
Merge: c980f52 00dd65f
Author: Stewart Smith <stewart@linux.vnet.ibm.com>
Date:   Wed Nov 8 14:23:08 2017 +1100

    Merge branch 'release' with op-build v1.20

Build system details:

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux ka1 4.9.0-0.bpo.2-amd64 #1 SMP Debian 4.9.18-1~bpo8+1 (2017-04-10) x86_64 GNU/Linux

cyrilbur-ibm commented 6 years ago

For what it worth this doesn't look like a XML parsing bug. Without delving too deeply into the code, it looks like getEnumerationType() wants the XML to have:

<enumerationType>
        <id>IBSCOM_MCS_BASE_ADDR</id>
        ....
</enumerationType>

Which I cannot find anywhere...

dcrowell77 commented 6 years ago

The problem is that it is falling down an else leg. Basically we can't interpret the value of the attribute as a number so we assume it must be an enumerated constant. In this case it is not an enumeration so we fail.

This is what I see in palmetto.xml

IBSCOM_MCS_BASE_ADDR 0x0003E00000000000,0x40000000000,0x10000000000,0x2000000000

So something in our xml parsing is turning that array value into a big string instead. This attribute is not defined as an array in attribute_types.xml so there is a disconnect between the system mrw and what hostboot expects. We handle this inside of processMrw.pl like this.

my ($base,$node_offset,$proc_offset,$offset) = split(/,/,
           $targetObj->getAttribute($target,"IBSCOM_MCS_BASE_ADDR"));
my $i_base = Math::BigInt->new($base);
my $i_node_offset = Math::BigInt->new($node_offset);
my $i_proc_offset = Math::BigInt->new($proc_offset);
my $i_offset = Math::BigInt->new($offset);
my $mcs = $targetObj->getAttribute($target, "MCS_NUM");
#Note: Hex convert method avoids overflow on 32bit machines
my $mcsStr=sprintf("0x%016s",substr((
     $i_base+$i_node_offset*$node+
     $i_proc_offset*$proc+$i_offset*$mcs)->as_hex(),2));
$targetObj->setAttribute($target, "IBSCOM_MCS_BASE_ADDR", $mcsStr);

So something in a perl update broke this logic such that it is no longer generating valid HB xml. Can you look in your output/host/usr/powerpc64le-buildroot-linux-gnu/sysroot/openpower_mrw_scratch/PALMETTO_hb.mrw.xml to see what xml was actually generated? That file is the output from processMrw.pl.

cyrilbur-ibm commented 6 years ago

github doesn't like "xml" so "txt" it is...

PALMETTO_hb.mrw.xml.txt

dcrowell77 commented 6 years ago

This is the problem:

    <attribute>
        <id>IBSCOM_MCS_BASE_ADDR</id>
        <default>0x000000000000000N</default>
    </attribute>

cyrilbur-ibm commented 6 years ago

Should I send a patch to remove those lines? Orrr?

dcrowell77 commented 6 years ago

Those lines are generated, there is nothing to patch. The problem is in the processMrw.pl somewhere where it is incorrectly processing the MRW xml, most likely something in the code snippet I pasted above.

duda-patryk commented 6 years ago

I had the same issue and changing line my $i_offset = Math::BigInt->new($offset); into my $i_offset = hex($offset); helped It turned out that sometimes i_offset is NaN

dcrowell77 commented 6 years ago

If someone has a working patch, feel free to make a pull request

shenki commented 6 years ago

I see this too. Latest op-build on Ubuntu 17.10 ppc64le

perl --version

This is perl 5, version 26, subversion 0 (v5.26.0) built for powerpc64le-linux-gnu-thread-multi

op-build$ grep -r 0x000000000000000N
output/host/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml:      <default>0x000000000000000N</default>
output/host/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml:      <default>0x000000000000000N</default>
output/host/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml:      <default>0x000000000000000N</default>
output/host/powerpc64le-buildroot-linux-gnu/sysroot/hostboot_build_images/temporary_hb.hb.xml:      <default>0x000000000000000N</default>

paulmenzel commented 6 years ago

Just a note, that it’s still present with Ubuntu 18.04 LTS.

open-power / op-build

Firestone PNOR build fails with "[FATAL!] Could not find enumeration with ID of IBSCOM_MCS_BASE_ADDR" #871

copy the firestone xml where the common lives

generate the system mrw xml

merge in any system specific attributes, hostboot attributes

creating the targeting binary