search

ARM-software / bsa-acs

Arm SystemReady : BSA Architecture Compliance Suite
Apache License 2.0
17 stars 42 forks source link

Integrate SBSA ACS into BSA ACS #101

Open hrw opened 1 year ago

hrw commented 1 year ago

Several tests from SBSA ACS moved to BSA ACS to the level when running sbsa.efi orders to run also BSA ACS.

So maybe it is time to merge two of them into one application?

One binary to test compliance with BSA spec, all levels of SBSA one and in future also other xBSA specs.

hrw commented 1 year ago

Also SBSA ACS info block is worse at some entries than same in BSA ACS:

 PE_INFO: Number of PE detected       :    2 
 GIC_INFO: Number of GICD             :    1 
 GIC_INFO: Number of ITS              :    1 
 TIMER_INFO: Number of system timers  :    0 
 WATCHDOG_INFO: Number of Watchdogs   :    1 
 CACHE_INFO: Number of cache nodes    :    5
 MPAM INFO: Number of MSC nodes       :    0 
 RAS2_INFO: Number of RAS2 entries    :    0 
 RAS2_INFO: Num of RAS2 memory entries:    0 
 PCIE_INFO: Number of ECAM regions    :    1 
 PCIE_INFO: Number of BDFs found      :    1
 PCIE_INFO: Number of RCiEP           :    1 
 PCIE_INFO: Number of RCEC            :    0 
 PCIE_INFO: Number of EP              :    0 
 PCIE_INFO: Number of RP              :    0 
 PCIE_INFO: Number of iEP_EP          :    0 
 PCIE_INFO: Number of iEP_RP          :    0 
 SMMU_INFO: Number of SMMU CTRL       :    0 
 Peripheral: Num of USB controllers   :    1 
 Peripheral: Num of SATA controllers  :    1 
 Peripheral: Num of UART controllers  :    1 
 PMU_INFO: Number of PMU units        :    0
 RAS_INFO: Number of RAS nodes        :    0
 PE_INFO: Number of PE detected       :    2 
 GIC_INFO: Number of GICD             :    1 
 GIC_INFO: Number of ITS              :    1 

 TIMER_INFO: Number of system timers  :    0 
 WATCHDOG_INFO: Number of Watchdogs   :    1 
 PCIE_INFO: Number of ECAM regions    :    1 
 PCIE_INFO: Number of BDFs found      :    10
 PCIE_INFO: Number of RCiEP           :    0 
 PCIE_INFO: Number of RCEC            :    0 
 PCIE_INFO: Number of EP              :    4 
 PCIE_INFO: Number of RP              :    5 
 PCIE_INFO: Number of iEP_EP          :    0 
 PCIE_INFO: Number of iEP_RP          :    0 
 SMMU_INFO: Number of SMMU CTRL       :    0 
 Peripheral: Num of USB controllers   :    1 
 Peripheral: Num of SATA controllers  :    1 
 Peripheral: Num of UART controllers  :    1

PCIe info:

Shell> pci
   Seg  Bus  Dev  Func
   ---  ---  ---  ----
    00   00   00    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 0008 Prog Interface 0
    00   00   01    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 10D3 Prog Interface 0
    00   00   02    00 ==> Display Controller - VGA/8514 controller
             Vendor 1234 Device 1111 Prog Interface 0
    00   00   03    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   04    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   05    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   06    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   07    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   01   00    00 ==> Mass Storage Controller - Non-volatile memory subsystem
             Vendor 1B36 Device 0010 Prog Interface 2
    00   02   00    00 ==> Serial Bus Controllers - USB
             Vendor 1B36 Device 000D Prog Interface 30
    00   03   00    00 ==> Display Controller - Other display controller
             Vendor 1234 Device 1111 Prog Interface 0
    00   04   00    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 10D3 Prog Interface 0
    00   05   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000E Prog Interface 0
    00   06   08    00 ==> Pre 2.0 device - UNDEFINED
             Vendor 1AF4 Device 1005 Prog Interface 0
chetan-rathore commented 1 year ago

Several tests from SBSA ACS moved to BSA ACS to the level when running sbsa.efi orders to run also BSA ACS.

So maybe it is time to merge two of them into one application?

One binary to test compliance with BSA spec, all levels of SBSA one and in future also other xBSA specs.

Hi @hrw,

Thanks for the feedback and we will discuss this internally.

The reason for separate BSA and SBSA binaries is due to BSA and SBSA being separate specification and their applicability. BSA goal is to ensure sufficient standard system architecture to enable OS boot. SBSA is built on top of BSA and has extra rules which are applicable to the Server.

Also from the SystemReady side, BSA is required across all the 3 SystemReady bands (IR, ES, SR), whereas SBSA is only required for SR. https://github.com/ARM-software/arm-systemready

Thanks, Chetan

chetan-rathore commented 1 year ago

Also SBSA ACS info block is worse at some entries than same in BSA ACS:

 PE_INFO: Number of PE detected       :    2 
 GIC_INFO: Number of GICD             :    1 
 GIC_INFO: Number of ITS              :    1 
 TIMER_INFO: Number of system timers  :    0 
 WATCHDOG_INFO: Number of Watchdogs   :    1 
 CACHE_INFO: Number of cache nodes    :    5
 MPAM INFO: Number of MSC nodes       :    0 
 RAS2_INFO: Number of RAS2 entries    :    0 
 RAS2_INFO: Num of RAS2 memory entries:    0 
 PCIE_INFO: Number of ECAM regions    :    1 
 PCIE_INFO: Number of BDFs found      :    1
 PCIE_INFO: Number of RCiEP           :    1 
 PCIE_INFO: Number of RCEC            :    0 
 PCIE_INFO: Number of EP              :    0 
 PCIE_INFO: Number of RP              :    0 
 PCIE_INFO: Number of iEP_EP          :    0 
 PCIE_INFO: Number of iEP_RP          :    0 
 SMMU_INFO: Number of SMMU CTRL       :    0 
 Peripheral: Num of USB controllers   :    1 
 Peripheral: Num of SATA controllers  :    1 
 Peripheral: Num of UART controllers  :    1 
 PMU_INFO: Number of PMU units        :    0
 RAS_INFO: Number of RAS nodes        :    0
 PE_INFO: Number of PE detected       :    2 
 GIC_INFO: Number of GICD             :    1 
 GIC_INFO: Number of ITS              :    1 

 TIMER_INFO: Number of system timers  :    0 
 WATCHDOG_INFO: Number of Watchdogs   :    1 
 PCIE_INFO: Number of ECAM regions    :    1 
 PCIE_INFO: Number of BDFs found      :    10
 PCIE_INFO: Number of RCiEP           :    0 
 PCIE_INFO: Number of RCEC            :    0 
 PCIE_INFO: Number of EP              :    4 
 PCIE_INFO: Number of RP              :    5 
 PCIE_INFO: Number of iEP_EP          :    0 
 PCIE_INFO: Number of iEP_RP          :    0 
 SMMU_INFO: Number of SMMU CTRL       :    0 
 Peripheral: Num of USB controllers   :    1 
 Peripheral: Num of SATA controllers  :    1 
 Peripheral: Num of UART controllers  :    1

PCIe info:

Shell> pci
   Seg  Bus  Dev  Func
   ---  ---  ---  ----
    00   00   00    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 0008 Prog Interface 0
    00   00   01    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 10D3 Prog Interface 0
    00   00   02    00 ==> Display Controller - VGA/8514 controller
             Vendor 1234 Device 1111 Prog Interface 0
    00   00   03    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   04    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   05    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   06    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   07    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   01   00    00 ==> Mass Storage Controller - Non-volatile memory subsystem
             Vendor 1B36 Device 0010 Prog Interface 2
    00   02   00    00 ==> Serial Bus Controllers - USB
             Vendor 1B36 Device 000D Prog Interface 30
    00   03   00    00 ==> Display Controller - Other display controller
             Vendor 1234 Device 1111 Prog Interface 0
    00   04   00    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 10D3 Prog Interface 0
    00   05   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000E Prog Interface 0
    00   06   08    00 ==> Pre 2.0 device - UNDEFINED
             Vendor 1AF4 Device 1005 Prog Interface 0

Hi @hrw,

Is your concern around the difference between the number of PCIe devices reported in BSA and SBSA?

That is due to BSA checking compliance of non-integrated devices (RP, EP) and SBSA checking compliance of integrated devices (RCiEP, iEP, iRP, RCEC). So the ACS is reporting some devices as 0 accordingly, but we can move those prints to DEBUG.

Thanks, ACS team

hrw commented 1 year ago

The reason for separate BSA and SBSA binaries is due to BSA and SBSA being separate specification and their applicability. BSA goal is to ensure sufficient standard system architecture to enable OS boot. SBSA is built on top of BSA and has extra rules which are applicable to the Server.

Now SBSA spec is built on top of BSA, was separate before (as it is older than BSA).

Many tests present in SBSA ACS moved to BSA one and now it is written "BSA ACS has to be run" at the end of SBSA ACS output.

One binary with "-sbsa -l X" would run BSA tests and SBSA level X ones. Listed one after another.

Also from the SystemReady side, BSA is required across all the 3 SystemReady bands (IR, ES, SR), whereas SBSA is only required for SR. https://github.com/ARM-software/arm-systemready

You forgot SystemReady LS ;D

hrw commented 6 months ago

Any update in 2024?

chetan-rathore commented 6 months ago

Hello @hrw,

We are working on this and changes are expected to be upstreamed by April month. Please note

We are also discussing on having mechanism to build a unified Sbsa binary with Bsa tests.

Thanks, ACS team

hrw commented 6 months ago

Why bother with two binaries?

We have "BSA.efi -sbsa" now. Change argument to take SBSA level as argument, add SBSA tests and done.

"BSA.efi -sbsa 6" would run BSA tests + SBSA level 6 tests. Interleaved with each other so we still have PE/Memory/GIC/PCIe/etc sections:

      *** Starting PE tests ***

Operating System View:
   1 : Check Arch symmetry across PE              : Result:  PASS
   2 : Check for number of PE                     : Result:  PASS
   3 : Check for AdvSIMD and FP support           : Result:  PASS
   4 : Check PE 4KB Granule Support               : Result:  PASS
   6 : Check Cryptographic extensions             : Result:  PASS
   7 : Check Little Endian support                : Result:  PASS
   8 : Check EL1 and EL0 implementation           : Result:  PASS
   9 : Check for PMU and PMU counters             : Result:  PASS
  10 : Check PMU Overflow signal                  : Result:  PASS
  11 : Check num of Breakpoints and type          : Result:  PASS
  12 : Check Synchronous Watchpoints              : Result:  PASS
  13 : Check CRC32 instruction support            : Result:  PASS
  14 : Check PAuth if implementation              : Result:  PASS

Hypervisor View:
  51 : Check EL2 implementation                   : Result:  PASS
  52 : Check Stage 2 4KB Granule Support          : Result:  PASS
  53 : Check Stage2 and Stage1 Granule match      : Result:  PASS
  54 : Check for PMU counters                     : Result:  PASS
  55 : Check VMID breakpoint number               : Result:  PASS

Platform Security View:
  76 : Check EL3 implementation                   : Result:  PASS

SBSA level 6:
S001 : Check PE Granule Support                   : Result:  PASS
S002 : Check for 16-bit ASID support              : Result:  PASS
S003 : Check AARCH64 implementation               : Result:  PASS
S004 : Check FEAT_LPA Requirements             
       S_L3PE_04
       Checkpoint --  1                           : Result:  SKIPPED
S005 : Check for RAS extension                    : Result:  PASS
S006 : Check DC CVAP support                
       S_L4PE_02
       Checkpoint --  2                           : Result:  SKIPPED
S007 : Check for 16-Bit VMID                      : Result:  PASS
S008 : Check for Virtual host extensions          : Result:  PASS
S009 : Support Page table map size change         : Result:  PASS
S010 : Check for addr and generic auth            : Result:  PASS
S011 : Check Activity monitors extension 
       Failed on PE -    0 for Level=  6 
       S_L5PE_04
       Checkpoint --  1                           : Result:  FAIL
S012 : Check for SHA3 and SHA512 support          : Result:  PASS
S013 : Stage 2 control of mem and cache           : Result:  PASS
S014 : Check for FEAT_NV2 support                 : Result:  PASS
S015 : Check MPAM PE Requirements        
       S_MPAM_PE
       Checkpoint --  2                           : Result:  SKIPPED
S016 : Check MPAM LLC Requirements                
       S_MPAM_PE
       Checkpoint --  2                           : Result:  SKIPPED
S017 : Check SPE if implemented                   
       Failed on PE -    0 for Level=  6          
       B_PE_17
       Checkpoint --  1                           : Result:  FAIL
S018 : Check Branch Target Support                
       Failed on PE -    0 for Level=  6          
       S_L6PE_02
       Checkpoint --  1                           : Result:  FAIL
S019 : Check Protect Against Timing Fault
       Failed on PE -    0 for Level=  6 
       S_L6PE_03
       Checkpoint --  1                           : Result:  FAIL
S020 : Check PMU Version Support         
       Failed on PE -    0 for Level=  6 
       S_L6PE_04
       Checkpoint --  1                       : Result:  FAIL
S021 : Check AccessFlag DirtyState Update     : Result:  PASS
S022 : Check Enhanced Virtualization Trap     : Result:  PASS
S023 : Check Speculation Restriction     
       Failed on PE -    0 for Level=  6 
       B_SEC_01
       Checkpoint --  1                           : Result:  FAIL
S024 : Check Speculative Str Bypass Safe          : Result:  PASS
S025 : Check PEs Impl CSDB,SSBB,PSSBB             : Result:  PASS
S026 : Check PEs Implement SB Barrier    
       Failed on PE -    0 for Level=  6 
       B_SEC_04
       Checkpoint --  1                       : Result:  FAIL
S027 : Check PE Impl CFP,DVP,CPP RCTX    
       Failed on PE -    0 for Level=  6 
       B_SEC_05
       Checkpoint --  1                       : Result:  FAIL

One tool to test everything.

sunnywang-arm commented 1 month ago

We can have one tool called xbsa to test everything by wrapping sbsa-acs, bsa-acs, and future xbsa-acs to achieve that goal that One tool to test everything, but I don't see a reason why we need to remove the existing capability of building the binaries separately.

It is straightforward to run specific tool for checking compliance with specific specification. Users won't need to figure out the complicated meaning behind the tool options. Ideally, users can run the tool without adding any options.