search

openbmc / openbmc-test-automation

Apache License 2.0
101 stars 92 forks source link

Feb Support: MFG phase 2 Test / Driver stability test - Witherspoon #308

Closed gkeishin closed 7 years ago

gkeishin commented 7 years ago

Place holder for time spent on test and debugging.. and tracking defects to closure

gkeishin commented 7 years ago

[Barreleye v1.99.1-55-g51ca97e ] Journal flooded with phosphor-gevent REST Dbus request

[Witherspoon v1.99.1-51-g947e967] org.openbmc.control.led.service: Main process exited, code=exited, status=234/MAKE_STARTER

[Barreleye v1.99.1-51-g947e967] org.openbmc.TimeManager.service: Failed with result 'core-dump'

gkeishin commented 7 years ago

https://github.com/openbmc/openbmc/issues/1059 https://github.com/openbmc/openbmc/issues/1060 https://github.com/openbmc/openbmc/issues/1061 https://github.com/openbmc/openbmc/issues/1062 https://github.com/openbmc/openbmc/issues/1063

gkeishin commented 7 years ago

https://github.com/openbmc/openbmc/issues/1066

gkeishin commented 7 years ago

https://github.com/openbmc/openbmc/issues/1069 https://github.com/openbmc/openbmc/issues/1070

gkeishin commented 7 years ago

https://github.com/openbmc/openbmc/issues/1072 https://github.com/openbmc/openbmc/issues/1073

gkeishin commented 7 years ago

Github# System Reason/Heading

openbmc/openbmc#1077 w37 proc stat btime persistent openbmc/openbmc#1074 w37 obmc-flash-init.service: Failed openbmc/openbmc#1073 w37 Host OS fails to Boot openbmc/openbmc#1072 w37 core-dump Phosphor BMC State Manager

Yesterday night build #504:

Github# System Reason/Heading

openbmc/openbmc#1069 w37 Bad chip id Error openbmc/openbmc#1070 w37 REST reboot hangs BMC

gkeishin commented 7 years ago

https://github.com/openbmc/openbmc/issues/1078

gkeishin commented 7 years ago

Already spent 1.8 story point worth time for code update, testing, opening defects and following up with developers..

gkeishin commented 7 years ago

Updated details of the test driver run https://github.com/openbmc/openbmc-test-automation/issues/220

**Driver Master Build Code update and Test Statistics**

            **WITHERSPOON - MNFG**

Current:

Previous:

OK - Code update + Basic capabilities Power on/off/Os boot/ reset GOOD - CT + OK + Os boot test

Date Build Test Result Defects
02 Feb 2017 v1.99.1-81-g42a851a OK
02 Feb 2017 v1.99.1-82-gbfd622e OK 1069 1070
03 Feb 2017 v1.99.1-94-g7d19e6c OK 1072 1073
03 Feb 2017 v1.99.1-96-g2a46570 OK 1078

NT - Not Tested CU - Code update ( BMC + PNOR) Note: For now PNOR update is Not applicable CT Test case count : 121 Note only Non poweron

Build CU Power On OS boot Standby Power Policy CT %
v1.99.1-81-g42a851a OK PASSED PASSED PASSED NT NT
v1.99.1-82-gbfd622e OK PASSED PASSED PASSED NT 91
v1.99.1-83-g82472ba OK PASSED PASSED PASSED NT 97.7
v1.99.1-94-g7d19e6c OK PASSED PASSED PASSED PASSED 100
v1.99.1-95-g807fdd2 OK PASSED PASSED PASSED PASSED 100
v1.99.1-96-g2a46570 OK PASSED PASSED PASSED FAILED NT

DEFECTS

No. Github# System Reason/Heading
1 openbmc/openbmc#1078 w37 Host OS starting Issue
2 openbmc/openbmc#1077 w37 Proc stat boot epoch time persistent
3 openbmc/openbmc#1074 w37 obmc-flash-init.service: Failed
4 openbmc/openbmc#1073 w37 Host OS fails to Boot
5 openbmc/openbmc#1072 w37 core-dump Phosphor BMC State Manager
6 openbmc/openbmc#1059 w37 Poweron Vcs workaround did not work properly
7 openbmc/openbmc#1060 w37 DBus.Error.Failed Time Manager
8 openbmc/openbmc#1061 w37 Job dev-aspeed-mbox.device 'timeout'
9 openbmc/openbmc#1062 w37 HMON warning filling up journal
10 openbmc/openbmc#1063 w37 Failed to create session org.freedesktop.login1
11 openbmc/openbmc#1066 w37 Host Sensor Property BootProgress shows Off
12 openbmc/openbmc#1067 w37 Service dead or inactive
13 openbmc/openbmc#1069 w37 Bad chip id Error
14 openbmc/openbmc#1070 w37 REST reboot hangs BMC
gkeishin commented 7 years ago

Current:

Previous:

gkeishin commented 7 years ago 
openbmc/openbmc#1097    w37     Reboot Fails
gkeishin commented 7 years ago

No new defects seen on new build https://openpower.xyz/job/openbmc-build/513/ as compared to last 2-3 builds test run

gkeishin commented 7 years ago

I could only verify only the code update https://openpower.xyz/job/openbmc-build/514/ and poweron and host OS boot test.. post that poooffs network gone.. So halted the end to end test for this build.

gkeishin commented 7 years ago

*** Test Outlook ***

BUILD_ID   : https://openpower.xyz/job/openbmc-build/distro=ubuntu,target=witherspoon/521/
VERSION_ID : v1.99.2-14-g38c1d04
BUILD TIME : 20170210024342
PNOR VERSION:

open-power-witherspoon-v1.14-dirty
        buildroot-6b9fd80
        skiboot-5.4.3-opdirty
        hostboot-edea301-opdirty
        linux-4.10-rc4-rc4-0c63710
        petitboot-v1.4.0-2284300
        witherspoon-xml-site_local-pragupta-671f883
        occ-4141b5f
        hostboot-binaries-674c635
        capp-ucode-1bb7503

BMC update  : PASSED
PNOR Update : NA
Reboot to Standby : PASSED  ( Multiple)
Boot test      :   PASSED        Host OS Booted  : YES
OS reboot Test :  sudo reboot --> boots back OS   ( FAILED ) 
CT - Non poweron ( Sensors and IPMI related failure due to #1107 / #1108 / #1109  )

New bugs :
  * OS reboot and BMC notification broken ( https://github.com/openbmc/openbmc/issues/1108 )
  * Inband IPMI failure on boot to standby ( https://github.com/openbmc/openbmc/issues/1109 )
  * ipmid fails to start (https://github.com/openbmc/openbmc/issues/1107 )
gkeishin commented 7 years ago

*** Test Outlook ***

SYSTEM     : W30
BUILD_ID   : https://openpower.xyz/job/openbmc-build/distro=ubuntu,target=witherspoon/528/
VERSION_ID : v1.99.2-24-g17064cb
BUILD TIME : 20170211042525
PNOR VERSION:

`pflash -r /tmp/out.txt -P VERSION`

open-power-witherspoon-v1.14-dirty
        buildroot-6b9fd80
        skiboot-5.4.3-opdirty
        hostboot-edea301-opdirty
        linux-4.10-rc4-rc4-0c63710
        petitboot-v1.4.0-2284300
        witherspoon-xml-site_local-pragupta-671f883
        occ-4141b5f
        hostboot-binaries-674c635
        capp-ucode-1bb7503

BMC update  : PASSED
PNOR Update : NA
Reboot to Standby : PASSED  ( Multiple)
Boot test      :   PASSED in W20 / Failed in W38        Host OS Booted  : NA
OS reboot Test :  sudo reboot --> boots back OS   ( NT )  Known bugs 
CT - Non poweron ( IPMI related failures due to IPMID failure )

New bugs :
  * https://github.com/openbmc/openbmc/issues/1115   ( Closed )
  * https://github.com/openbmc/openbmc/issues/1116   
  * https://github.com/openbmc/openbmc/issues/1118
gkeishin commented 7 years ago

*** Driver build Test Outlook - Witherspoon ***

SYSTEM     : W37 ( Witherspoon)
BUILD_ID   : https://openpower.xyz/job/openbmc-build/distro=ubuntu,target=witherspoon/535/
VERSION_ID : v1.99.2-32-g71526c2
BUILD TIME : 20170214025621
PNOR VERSION:

`pflash -r /tmp/out.txt -P VERSION`

open-power-witherspoon-v1.14-dirty
        buildroot-6b9fd80
        skiboot-5.4.3-opdirty
        hostboot-edea301-opdirty
        linux-4.10-rc4-rc4-0c63710
        petitboot-v1.4.0-2284300
        witherspoon-xml-site_local-pragupta-671f883
        occ-4141b5f
        hostboot-binaries-674c635
        capp-ucode-1bb7503

BMC update  : PASSED
PNOR Update : NA
Reboot to Standby : PASSED  ( Multiple)
Boot test      :   PASSED        Host OS Booted  : YES
OS reboot Test :  sudo reboot --> boots back OS   ( NT )  Known bugs ( refer Witherspoon report https://github.com/openbmc/openbmc-test-automation/issues/220)
CT - Non poweron ( IPMI related failures due to IPMID failure )

New bugs :
         * None
gkeishin commented 7 years ago

*** Driver build Test Outlook - Witherspoon ***

SYSTEM     : W38 ( Witherspoon)
BUILD_ID   : https://openpower.xyz/job/openbmc-build/distro=ubuntu,target=witherspoon/543/
VERSION_ID : v1.99.2-46-g04f01f7
BUILD TIME : 20170215035323
PNOR VERSION:

`pflash -r /tmp/out.txt -P VERSION`

open-power-witherspoon-713df6c-dirty
        buildroot-6b9fd80
        skiboot-8c51fda-opdirty
        hostboot-eb217e5-opdirty
        linux-4.9.4-opdirty-bdade6f
        petitboot-v1.4.0-opdirty-5ad2029
        witherspoon-xml-site_local-wghoffa-3a71790-dirty
        occ-4141b5f-opdirty
        hostboot-binaries-01cbe09-opdirty
        capp-ucode-1bb7503-opdirty

Test Ran

BMC update  : PASSED
PNOR Update : PASSED
Reboot to Standby : PASSED  ( Multiple)
Boot test      :   PASSED        Host OS Booted  : YES
OS reboot Test :  sudo reboot --> boots back OS   ( Known bug )  
CT - Non poweron ( NT )

PNOR updated with William Hoffa image

System : w33 PGOOD failure - OS didn't boot
       : w38 OS booted 

New bugs :
         * https://github.com/openbmc/openbmc/issues/1129
         * https://github.com/openbmc/openbmc/issues/1130
gkeishin commented 7 years ago

22 openbmc/openbmc#1132 w38 Kernel panic - not syncing:

gkeishin commented 7 years ago

*** Driver build Test Outlook - Witherspoon ***

SYSTEM     : W38/W30 ( Witherspoon)
BUILD_ID   : https://openpower.xyz/job/openbmc-build/distro=ubuntu,target=witherspoon/553/
VERSION_ID : v1.99.2-66-g1488f75
BUILD TIME : 20170217003621
PNOR VERSION:

`pflash -r /tmp/out.txt -P VERSION`

open-power-witherspoon-713df6c-dirty
        buildroot-6b9fd80
        skiboot-8c51fda-opdirty
        hostboot-eb217e5-opdirty
        linux-4.9.4-opdirty-bdade6f
        petitboot-v1.4.0-opdirty-5ad2029
        witherspoon-xml-site_local-wghoffa-3a71790-dirty
        occ-4141b5f-opdirty
        hostboot-binaries-01cbe09-opdirty
        capp-ucode-1bb7503-opdirty

Test Ran

BMC update  : PASSED
PNOR Update : PASSED
Reboot to Standby : PASSED  ( Multiple)
Boot test      :   PASSED        Host OS Booted  : YES ( w30 )
OS reboot Test :  sudo reboot --> boots back OS   ( Known bug )  
CT - Non poweron ( NT )

New bugs on this driver :
    * https://github.com/openbmc/openbmc/issues/1157

On w38 / w33  OS didnt boot.
gkeishin commented 7 years ago

*** Driver build Test Outlook - Witherspoon ***

SYSTEM     : W38 ( Witherspoon)
BUILD_ID   : witherspoon-20170218231615.all.tar  ( openbmc master + openFSI enablement for MFG )
VERSION_ID : v1.99.2-71-gbc49f79
BUILD TIME : 20170218231615
PNOR VERSION:

`pflash -r /tmp/out.txt -P VERSION`

IBM-witherspoon-ibm-OP9_v1.14_1.4
        op-build-5f2e132
        buildroot-6b9fd80
        skiboot-8c51fda
        hostboot-6782606
        linux-4.9.8-10d96bf
        petitboot-v1.4.1-42f37cc
        witherspoon-xml-3fd670a
        occ-4141b5f
        hostboot-binaries-541a2bd
        capp-ucode-1bb7503

Test Ran


BMC update  : Already updated
PNOR Update : PASSED
Reboot to Standby : PASSED
Poweron IPL test  : PASSED
Host OS Booted    : YES
Power On IPL test( PowerOn / Poweroff with OS boot) :  Iteration 3  - PASSED
OS reboot Test    :  sudo reboot --> boots back OS :  ( Not fixed )
CT - Non poweron  : PASSED ( 146  TC )
gkeishin commented 7 years ago

Master Jenkins BMC + (MFG) PNOR Build Test for MFG2

- [x] SYSTEM     : W37 ( Witherspoon)
- [x] BUILD_ID   : https://openpower.xyz/job/openbmc-build/564/distro=ubuntu,target=witherspoon/
- [x] VERSION_ID : v1.99.2-89-gf327ed7
- [x] BUILD TIME : 20170221021702
- [x] PNOR VERSION:

`pflash -r /tmp/out.txt -P VERSION`
IBM-witherspoon-ibm-OP9_v1.14_1.4
        op-build-5f2e132
        buildroot-6b9fd80
        skiboot-8c51fda
        hostboot-6782606
        linux-4.9.8-10d96bf
        petitboot-v1.4.1-42f37cc
        witherspoon-xml-3fd670a
        occ-4141b5f
        hostboot-binaries-541a2bd
        capp-ucode-1bb7503

Test Ran

- [x] BMC update        : PASSED
- [x] PNOR Update       : NA
- [x] Reboot to Standby : PASSED   ( 2 )
- [x] Poweron IPL test  : PASSED   ( 3 )   (ON/OFF sequence)
- [x] Host OS Booted    : YES
- [x] OS Reboot Test    : NA ( Not fixed - so not tested )
- [x] CT - Non poweron  : PASSED ( 146 TC )
- [x] Inventory         : PASSED ( Data populated on IPL )
      Persistency       : NO ( Expected - Not Persistant across resets )
                 [Note] : Version field data issue ( Looks like its its not yet fixed )
- [x] eSEL              : NA  ( Insufficient privilege level )
      Persistency       : NA ( Expectation is to persist across resets )
      REST              : NA ( Expectation is to see the eSEL from REST Get view )
                 [Note] : ipmitool privilege level is not raised for eSEL generation

New bugs on this driver :
    * None