osflow setup readme is out of date/broken

[jeremyherbert]

In the osflow setup, the guide is out of date. The first step is to run make, but this generates an error because the makefiles do not set TOP, BOARD_SRC and DESIGN_SRC.

I am referring to the readme here: https://github.com/stnolting/neorv32/tree/master/setups/osflow

[stnolting]

In the osflow setup, the guide is out of date.

Thanks for the hint! :+1: I will rework that.

[umarcor]

Let me break the following down: make BOARD=Fomu TASK='clean load' MinimalBoot. In order to generate a bitstream, we need the following elements:

1. RTL sources of the core.
2. RTL sources of the memories (imem and/or dmem).
3. RTL sources of the ProcessorTop, and maybe SystemTop.
4. RTL of the BoardTop, and maybe other specific sources.
5. Constraints.
6. Components of the hard IP in the device.

So, we handle them as follows:

• When calling make in setups/examples, you need to set BOARD, at least, and you need to use the design name as the target.
• The targets of the Makefile which are a Design name will set the DESIGN variable, will provide the specific sources for that design (3.) and will call the Makefile again, targeting the BOARD name.
• Therefore, when called internally targeting a board, the following variables need to be defined (by the previous step): DESIGN, DESIGN_SRC, BOARD and optionally NEORV32_VERILOG_SRC. Then, the target BITSTREAM variable will be set, and it will call itself again, targeting run. This stage, specific to the board, is where 2. can be overriden through variable NEORV32_MEM_SRC (which needs to contain one RTL for imem and another one for dmem).
• When called with target run, the BOARD_SRC variable (4.) will be set automatically using the BOARD and DESIGN variables set in previous calls (that should correspond the top-level VHDL source in setups/examples/). The TOP will also be set, and the ID (which is the appendix of the bitstream) will be set to DESIGN. Then, the makefile in setups/osflow/$(BOARD) will be executed, targeting TASK. TASK is, by default, clean $(BITSTREAM).
• So, all of setups/osflow/$(BOARD)/Makefile have a common structure. The only differences in the board specific Makefiles are the following:
  • PCF: pointing to the constraints file (5.). See hdl/constraints.
  • PNRFLAGS: containing the arguments for nextpnr (package, device, etc.).
  • ID: default appendix to the bitstream (overriden when using the setup/examples/Makefile).
  • IMPL: default name of the bitstream (which should match what is defined in setup/examples/Makefile). Apart from that, all of the board Makefiles do import filesets.mk (where 1. and 6. are defined), they import tools.mk, synthesis.mk and PnR_Bit.mk (where the common tool calls are defined for analysis, synthesis, implementation and bitstream generation). Some Makefiles do have some additional target for programming the board (say load).

In the example command I provided above, I override the TASK and set it to clean load. Hence, instead of just generating the bitstream, it will also program the board.

Now, the reason that is not properly documented is because I'm not confident about it. As I proposed @stnolting, I knew the hierarchy of VHDL sources we wanted to have, due to the abstractions that BoardTop, SystemTop and ProcessorTop provide, and how they allow reusability across boards and toolchains. Then, I picked the Makefile based solution from antonblanchard/microwatt, ghdl/ghdl-yosys-plugin and im-tomu/fomu-workshop (which are all alike) and I applied them to the use case here. After that, it escalated quite quickly. UPduino, Fomu, iCESugar, TinyFPGA-BX, using a similar structure in Vivado, Quartus and Radiant toolchains. So we should now rethink it.

• It feels unintuitive to use the design as a target, and setting variable TASK for actually deciding what to do. It would be more natural to call make BOARD=iCESugar DESIGN=MinimalBoot clean load or make BOARD=iCESugar DESIGN=MinimalBoot bitsream (which was one of Jeremy's attempts in #97, although using a different parameter order).
• As soon as we support ECP5, people will use it, since it's a rather common device family. Supporting ECP5 is actually pretty straightforward, since the toolchain is almost the same as for ICE40. However, some tweaks need to be made (mainly an ECP5_SRC variable in filesets.mk, and maybe other default NEORV32_MEM_SRC). Unfortunately, I don't have any ECP5 board with me, so I'd need help from someone to test that.
• We support two different toolchain types and we will soon support three:
  • 1. A single environment with all the tools with GHDL built into Yosys.
  • 2. A single environment with all the tools with GHDL built as a module.
  • 3. Multiple environments (containers) each with a tool, with GHDL built as a module.
• We are calling a makefile, which does at least two additional calls to itself and then has at least 2 other levels of calls into the partial makefiles. Having at least 5 levels in the stack seems to be hitting the limit of the usability of Makefiles. They do work, but debugging is painful, very specially for the newcomers who are not familiar with the structure.
  • Something I didn't mention explicitly is that the filenames that are defined in the variables of osflow/examples/Makefile need to specify the relative location taking the board specific Makefile as a root, not the examples folder. That is not a problem for using the Makefiles, but it is very confusing for contributors (as @juanmard suffered).

Overall, I think it is time to seriously consider Python for better management of structures with fields, strings, paths and errors. I do know that @stnolting is not a big fan of Python. However, as I argued in previous issues, it is unavoidable at some point of complexity to use a proper programming/scripting language which is beyond just automating some tasks. Don't take me wrong, Makefiles and bash are good, and with some effort we could go forward with them. However, for string management and for dealing with structures, they fall short, and we need to rely on naming of variables which increases the maintenance burden as we add boards.

Naturally, this is not some brilliant idea of mine. This was the conclusion of several members of the community during the last decades. Currently, there are about a dozen projects written in Python for this specific task. And some other in other languages. Unfortunately, all of those are isolated towers, and there is little cooperation/coordination between them. I wrote about the topic in Open Source Verification Bundle: API | Core and Open Source Verification Bundle: API | Tool. Find the bigger picture in bit.ly/unai-eda-megadoc. Lately, @rodrigomelo9 has been prototyping PyFPGA/symbiflow_cli. I pinged him in #83. However, and this is a big however, there is additional complexity related to using any of those existing Python projects:

On the one hand, it makes sense to use them instead of reinventing the wheel. We do NOT want to create a multitool, multiboard, mixed-language build system ourselves, because that is not the scope of this project and we have much better things to waste our time on.

On the other hand, I'm not sure about any of those projects being currently able to support the use cases needed here. This is, in fact, my motivation for using Makefiles in the first place. Not only for initially pleasing @stnolting (which also), but because I don't have a solid alternative which I can strongly recommend. Explicitly:

• We use VUnit for testing and we want to progressively enhance that infrastructure for using more advanced Verification Components and strategies. VUnit is a build and project manager for testing only, synthesis and implementation are out of scope. However, most of the files defined in the run.py are used for synthesis too, indeed. Hence, we might want to use the same "source of truth" for defining the fileset. That is a problem with several of the tools. Either VUnit is not supported, or the run.py is autogenerated, making the project management tool necessary for simulation too, or forcing duplicated filesets/sources.
• Managing HDL only projects is well supported. So, osflow, quartus and radiant examples might be supported. However, I did not see any of those tools used for Vivado projects including Board Design components, typically saved as TCL plus an instantiation template. We commented about that in https://github.com/stnolting/neorv32/discussions/52. It seems relatively easy to do using the GUI, but not programmatically. I'm interested in that use case for targeting Zynq boards, such as Arty-Z7, PYNQ, etc. There, a single-core or dual-core ARM is included, and the NEORV32 can be used as an orchestrator on the PL side, which communicates with a Linux running on the ARM through AXI. Alternatively, an HDL instantiation of all the PS and interconnects might be used, but that is difficult for users to extend/customise.
• I'm not sure about any of the projects supporting VHDL with all the toolchains: osflow (MSYS2, all-in-one containers, fine-grained contaienrs, native), quartus, radiant, vivado and symbiflow. I know that PyFPGA and FuseSoC/edalize support most of them, but not sure about all.

Therefore, my very personal proposal would be to use pyCAPI, as argued in Open Source Verification Bundle: API | Core and as I showcased with VUnit in umarcor/osvb: AXI4Stream/AXI4Stream.core and umarcor/osvb: AXI4Stream/test/vunit/run_capi.py (compare to a regular VUnit script: umarcor/osvb: AXI4Stream/test/vunit/run.py). Now, you will understand that the structure I used in the partial makefiles of this project was not arbitrary (see setups/osflow/filesets.mk and setups/vivado/arty-a7-test-setup/create_project.tcl). I did that in preparation for supporting *.core files, either the current official format supported by FuseSoC, or my pyCAPI proposal for having it split and be reusable for other projects. The filesets should be equally reusable in any other project management tool.

Nonetheless, I do understand my position is very opinionated. I absolutely understand if any other approach is preferred, since this is a rather gray area with no obvious best choice. In some other issues, I commented that I would be ok with having multiple of those projects used here (i.e. have a configuration file for FuseSoC, another one for PyFPGA, another one for tsfpga, etc.), as long as someone else is willing to take care of each of those. Unfortunately, I can help maintain CI, keep an eye on VUnit and help otherwise, but I don't have bandwidth for learning the details of those projects.

[jeremyherbert]

It's interesting that you mention an ECP5 port, as that is what I was going to try and do once I have my head around how things work in this repository. I have a few boards here (orangecrab, ULX3S and a lattice dev kit) to test with. If you have anything you'd like to test, please let me know.

If you think that the opinion of a random outsider is useful, I do agree that some sort of python tooling around edalize would be the way forward. This is especially relevant given that there is more open source PnR being worked on (SymbiFlow/prjxray, and Ravenslofty/mistral and that will almost certainly be usable through edalize in due course.

[stnolting]

@umarcor Thanks for the great explanation and recap! 👍 ❤️

@jeremyherbert

If you think that the opinion of a random outsider is useful, I do agree that some sort of python tooling around edalize would be the way forward.

As @umarcor already said, I am no big fan of python (yet?). But I see the problems here. So maybe it is about time to move forward to a more sophisticated approach.

[umarcor]

It's interesting that you mention an ECP5 port, as that is what I was going to try and do once I have my head around how things work in this repository. I have a few boards here (orangecrab, ULX3S and a lattice dev kit) to test with. If you have anything you'd like to test, please let me know.

@jeremyherbert, let's start with the OrangeCrab. See #98.

If you think that the opinion of a random outsider is useful, I do agree that some sort of python tooling around edalize would be the way forward.

I am also a random outsider in this project! I contributed in several areas during the last weeks, but the core and the majority of the work is a single-man show (by @stnolting).

This is especially relevant given that there is more open source PnR being worked on (SymbiFlow/prjxray, and Ravenslofty/mistral and that will almost certainly be usable through edalize in due course.

Note that supporting multiple devices families is not specially complex if all of them are implemented in nextpnr. The modifications in #98 can be used for mistral, apicula, xray, uray, bureau, etc. I think the complexity relies on supporting multiple PnR tools: nextpnr, VTR, Vivado, Quartus... Most of the projects, do not support all of them. I'm not sure about edalize supporting the two following flows:

• VHDL -> GHDL -> Vivado.
• VHDL -> GHDL -> Yosys -> SymbiFlow (VTR).

Thanks for the great explanation and recap!

@stnolting, note that it can change significantly, as seen in #98. That's why I'm still unsure about writing all those details in a README (as suggested in https://github.com/stnolting/neorv32/commit/1c88f0382f30a24b6a2b2f1360ef5dfd21fb56cd#r53024192).

[stnolting]

Just for the records: Being a "project outsider" is no bad think at all! :wink: Especially not if you want to run one of the simple "hello world setups". This project aims to be something like a ready-to-use IP core. There is no need to know things like "how program counter updates are implemented in logic" as you can use the processor as black box: the interfaces (Wishbone, SPI, UART, ..., and even the stream links) are all based on common protocol specs.

Being "a project outsider" is also a good think for me, because sometimes I'm too focused on the inner workings of the core that I am forgetting about the world around it 😅

@umarcor

@stnolting, note that it can change significantly, as seen in #98. That's why I'm still unsure about writing all those details in a README (as suggested in 1c88f03#r53024192).

Oh it's ok. We will see what we come up with. Anyway, thanks for your efforts here! 👍