Provided Hello World Example Failing Test

[tianrui-wei]

Hi,

Thank you for open sourcing this work. I ran into a few problems during following the README and wonder if you could help me figure out what I was missing. Here's a short list of the steps I took.

1. I generated the bitstream for the hello_world example after sourcing the setup_hdk.sh
2. I installed the sdk after sourcing setup_sdk.sh
3. I created a bitstream in $CL/build/bitstreams using the script you provided
4. I compiled the hello world software in $CL/software/runtime
5. I installed the patched xdma kernel module in the sdk like described
6. I programmed the attached u200 with the generated bitstream and performed pci hot reset. After this step, my lspci output is

   01:00.0 Memory controller: Amazon.com, Inc. Device 1042 (rev ab)
   01:00.1 Memory controller: Amazon.com, Inc. Device 1041 (rev ab)

7. I ran the setpci script you provided to set the registers
8. I ran the test_hello_world binary with sudo privileges

Here's the output I got

reached fpga_mgmt_init point
===== Starting with peek_poke_example =====
Writing 0xefbeadde to HELLO_WORLD register (0x0000000000000500)
=====  Entering peek_poke_example =====
register: 0xffffffff
TEST FAILEDResulting value did not match expected value 0xdeadbeef. Something didn't work.
Developers are encouraged to modify the Virtual DIP Switch by calling the linux shell command to demonstrate how AWS FPGA Virtual DIP switches can be used to change a CustomLogic functionality:
$ fpga-set-virtual-dip-switch -S (slot-id) -D (16 digit setting)

In this example, setting a virtual DIP switch to zero clears the corresponding LED, even if the peek-poke example would set it to 1.
For instance:
# sudo fpga-set-virtual-dip-switch -S 0 -D 1111111111111111
# sudo fpga-get-virtual-led  -S 0
FPGA slot id 0 have the following Virtual LED:
1010-1101-1101-1110
# sudo fpga-set-virtual-dip-switch -S 0 -D 0000000000000000
# sudo fpga-get-virtual-led  -S 0
FPGA slot id 0 have the following Virtual LED:
0000-0000-0000-0000
tianruiw@jura:~/research/aws-fpga-f1-u200/hdk/cl/examples/cl_hello_world/software/runtime$ sudo fpga-set-virtual-dip-switch -S 0 -D 1111111111111111
tianruiw@jura:~/research/aws-fpga-f1-u200/hdk/cl/examples/cl_hello_world/software/runtime$ sudo fpga-get-virtual-led  -S 0
FPGA slot id 0 have the following Virtual LED:
0000-0000-0000-0000

I'd be really grateful if you could provide insight on why things went wrong.

Best Regards, Tianrui

[kyyalama2]

Dear customer,

Thank you so much for trying our F1-U200 developer kit. Can you please provide information on 1.) Tool version you used for the flow 2.) Confirm that you programmed the FPGA using JTAG?

Thanks

[kyyalama2]

In addition, can you also provide lspci dump using: lspci -vvv -s 0000:01:00.0 lspci -vvv -s 0000:01:00.1

[tianrui-wei]

Hi,

Thank you for your help.

1. I used vivado 2021.2 for the flow
2. I indeed programmed the FPGA using JTAG
3. Here are the lspci output

~$ sudo lspci -vvv -s 0000:01:00.0
[sudo] password for tianruiw:
01:00.0 Memory controller: Amazon.com, Inc. Device 1042 (rev ab)
    Subsystem: Device fedc:1d51
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Region 0: Memory at a0000000 (32-bit, non-prefetchable) [virtual] [size=32M]
    Region 1: Memory at a2000000 (32-bit, non-prefetchable) [virtual] [size=2M]
    Region 2: Memory at 6012000000 (64-bit, prefetchable) [virtual] [size=64K]
    Region 4: Memory at <unassigned> (64-bit, prefetchable) [virtual]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] MSI-X: Enable- Count=32 Masked-
        Vector table: BAR=2 offset=00008000
        PBA: BAR=2 offset=00008fe0
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s (ok), Width x16 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+, NROPrPrP-, LTR-
             10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS-, TPHComp-, ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [1c0 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
        LaneErrStat: LaneErr at lane: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
    Capabilities: [480 v0] Extended Capability ID 0x1d0f
    Kernel driver in use: xocl
    Kernel modules: xocl

~$ sudo lspci -vvv -s 0000:01:00.1
01:00.1 Memory controller: Amazon.com, Inc. Device 1041 (rev ab)
    Subsystem: Device fedc:1d50
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Region 0: Memory at 6000400000 (64-bit, prefetchable) [size=16K]
    Region 2: Memory at 6000404000 (64-bit, prefetchable) [size=16K]
    Region 4: Memory at 6000000000 (64-bit, prefetchable) [size=4M]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s (ok), Width x16 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+, NROPrPrP-, LTR-
             10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS-, TPHComp-, ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn-
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [480 v0] Extended Capability ID 0x1d0f

[kyyalama2]

Dear customer,

sorry for the delayed reply. We expect to see 128G for BAR4 and its possible that your host BIOS is not configured to enable big BAR. Can you please try and see if you can enable big BAR in the host BIOS setting and successfully enumerate BAR4

Thanks

[tianrui-wei]

Hi,

Thanks for your response. I checked the output of journalctl more closely, and I spotted the following lines

Jun 01 06:04:28 jura kernel: pci 0000:01:00.0: Max Payload Size set to 256 (was 128, max 1024)
Jun 01 06:04:28 jura kernel: pci 0000:01:00.0: BAR 4: no space for [mem size 0x2000000000 64bit pref]
Jun 01 06:04:28 jura kernel: pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x2000000000 64bit pref]

My understanding is that this space exceeds the space allocated by bios, thus making it fail? However, we have enabled above 4G decoding in UEFI. It's not clear to us what we can do to continue to make forward progress.

1. Perhaps you'd have some magical commands for generating an .mcs file for the aws F1 shell?
2. Is there any way we could resolve this problem?

Best Regards, Tianrui

[kyyalama2]

Dear customer, The IP in the repo requires some settings updates to be able to support MCS generation flow. We are aware of this and are working on providing these settings updates

Thanks

[tianrui-wei]

Thanks Kishore, I look forward to your subsequent commits to the repo!

[kyyalama2]

Hi Tianrui

1.) The MCS file generation is released to the repo 2.) Also noted that in the example setpcie script provided setpci -v -s 0000:c1:00.0 0x4.w=0x7 $ setpci -v -s 0000:c1:00.1 0x4.w=0x7 We used 0000:c1:00.0 However, as I see from your logs, your lspci shows 01:00.0 Memory controller: Amazon.com, Inc. Device 1042 (rev ab) 01:00.1 Memory controller: Amazon.com, Inc. Device 1041 (rev ab)

So just wanted to make sure, you replaced c1:00 with 01:00 in your setup. Can you please confirm if you did update the setpci accordingly to work with your bdf

Thanks

[tianrui-wei]

Hi Kishore,

Yes, I can confirm I updated the setpci commands accordingly.

Thanks for support the mcs flow so quickly, I'll keep you posted how it goes.

Best Regards, Tianrui