VCU118 Edge PCIe doesn't seem functional

It's possible I set this up wrong. I've removed the MSI interrupts from the overlay. I'm using Chipyard's generator to generate my harness. I've placed the overlay which I've customized a little bit. The only "custom" part is that the overlayoutput doesn't return the interrupt node.

I've added two mmio ports and a slave port to the rocket chip system. I'm using Chipyard's SD bootrom. I don't see any activity on the bootrom when I hook up the overlay.

Here's my harness:

class PCIeVCU118FPGATestHarness(override implicit val p: Parameters) extends VCU118FPGATestHarness {
    val custom_edge = Overlay(CustomPCIeOverlayKey, new CustomPCIeVCU118EdgeShellPlacer(this, CustomPCIeShellInput()))

    // TODO: Overlay is set to default to 4 lanes. We only need two. 

    val overlayOutput = dp(CustomPCIeOverlayKey).last.place(CustomPCIeDesignInput(wrangler=dutWrangler.node, corePLL=harnessSysPLL)).overlayOutput
    // val (pcieNode: TLNode, pcieIntNode: IntOutwardNode) = (overlayOutput.pcieNode, overlayOutput.intNode)
    val pcieNode : TLNode = overlayOutput.pcieNode 
    val (pcieSlaveTLNode: TLIdentityNode, pcieMasterTLNode: TLAsyncSinkNode) = (pcieNode.inward, pcieNode.outward)

    val outParamsl2frontend = topDesign match { case td: ChipTop =>
        td.lazySystem match { case lsys: CanHaveSlaveTLPort =>
              lsys.l2FrontendTLNode.edges.out(0) 

        }
    }
    val inParamsMMIOPeriph = topDesign match { case td: ChipTop => 
        td.lazySystem match { case lsys: CanHaveMasterTLMMIOPort => 
              lsys.mmioTLNode.edges.in(0)
        }
    }
    val inParamsControl = topDesign match {case td: ChipTop => 
        td.lazySystem match { case lsys: CanHaveMasterTLCtrlPort => 
            lsys.ctrlTLNode.edges.in(0)
        }
    } // There's probably a way to grab all these parameters in one go. 

    // (interrupts, _)
    // No interrupt in design for now 
    // val intParams = topDesign match { case td: ChipTop => 
    //     td.lazySystem match { case lsys: HasSyncExtInterrupts => 
    //         lsys.extInterrupts.edges.out(0)
    //         // ibus.fromSync
    //     }
    // }

    val pcieClient = TLClientNode(Seq(inParamsMMIOPeriph.master))
    val pcieCtrlClient = TLClientNode(Seq(inParamsControl.master))
    val connectorNode = TLIdentityNode()

    connectorNode := pcieClient 
    connectorNode := pcieCtrlClient 
    pcieSlaveTLNode :=* connectorNode  // Could be a need for overlayOutput.TLIn(pcieSlaveTLNode)

    val pcieManager = TLManagerNode(Seq(outParamsl2frontend.slave))
    pcieManager := pcieMasterTLNode  

    override lazy val module = new PCIeVCU118FPGATestHarnessImp(this)
}

And here's my config:

class CustomRocketConfig extends Config(
  new WithPCIeMMIOPort ++  // add default external master port
  new WithControlPort ++ // add control port for pcie cfg. 
  new freechips.rocketchip.subsystem.WithDefaultSlavePort ++ // add default external slave port
  new freechips.rocketchip.subsystem.WithNBigCores(2) ++
  // new freechips.rocketchip.subsystem.WithNExtTopInterrupts(3) ++
  new chipyard.config.AbstractConfig)

class WithPCIeTweaks extends Config (
    new WithPCIeClient ++ 
    new WithPCIeManager ++
    new WithPCIeCtrlClient ++  // Same for these harness binders - ME
    new WithPCIeIOBinders 
)

class WithPCIeIOBinders extends Config (
    new WithXDMAMasterIOPassthrough ++ 
    new WithXDMASlaveIOPassthrough ++ 
    new WithXDMACtrlIOPassthrough  // I imagine these three IOBinders can be combined into one - ME
)

class PCIeRocketConfig extends Config (
    new WithPCIeTweaks ++ 
    new WithVCU118Tweaks ++
    new WithCustomVCU118System ++ 
    new CustomRocketConfig 
)

Compilation seems fine. The .graphml shows all the connections are being made between the system and ports. The ports corresponding outer master/slave seem to connect fine to the XDMA overlay. Yet the core fails to show any activity it normally does through the bootrom UART print statements and there seems to be no activity on the PCIe edge when analyzed with a protocol analyzer. I've used the default XDMA configuration that configures it as a root port and tried configuring it as an endpoint instead. The result seems to be the same. I can try using an ILA to analyze the state of the fpga further. I can try isolating the issue by testing this separately from the chipyard system.

sifive / fpga-shells

VCU118 Edge PCIe doesn't seem functional #165