Closed MidsummerNight closed 2 years ago
mithro
commented
2 years ago Take a look at @bunnie's previous work at https://github.com/SymbiFlow/prjxray/pull/1155
The biggest issue is that you need a new enough Vivado that supports Spartan 7 but https://github.com/SymbiFlow/prjxray/issues/14 prevents us moving to a newer version than Vivado 2017.2
MidsummerNight
commented
2 years ago @mithro Thanks, but I've yet to reach the fuzzing steps, I'm stuck at update_parts.py and update_resources.py not being able to generate a correct parts.yaml for spartan7. I suspect something is wrong with my Makefile but I cannot find out what's wrong. @MaksRawski did manage to proceed to fuzzy testing in #1534 , but his repo is pretty outdated, his Makefile doesn't have the db-prepare-<database_name> target so I'm lost.
I'll try to manually write that parts.yaml for now to see it it works.
And why is Vivado's version an issue in my case? 2017.2 does support xc7s50t after all, so while we can't test other Spartan-7 chips without solving #14, we can still start with xc7s50t, right?
MidsummerNight
commented
2 years ago Oops my bad, the contents of /prjxray/settings/spartan7/devices.yaml should be:
# device to fabric mapping
"xc7s50":
fabric: "xc7s50"NOT:
# device to fabric mapping
"xc7s50t":
fabric: "xc7s50t"Notice the extra t, ugh.
Now the output is:
me@me-my_machine:~/prjxray$ make db-prepare-spartan7
Preparing spartan7 files
============================
Find pins for xc7s50csga324-1
Find pins for xc7s50fgga484-1
Find pins for xc7s50ftgb196-1
me@me-my_machine:~/prjxray$
with the correct parts.yaml generated.
I'll proceed and continue to report my progress/issues in this thread. After so many stagnated trials to support Spartan-7, let's hope this time we can really get it done!
bunnie
commented
2 years ago I have an interest in this too, but a lack of time. I'll keep an eye on this and if somehow I can be of assistance I'll try to help out.
MidsummerNight
commented
2 years ago Update: 000, 001, 005, 007 completed successfully. Do I just run 010- 101 sequentially or is there any order I should follow? The Guide says I should complete 000, 001, 005, 072, 073, 074, and 075 first, but fuzzers/Makefile shows a different order. I'll follow the guide and try 072, 073, 074, 075 for now and see.
acomodi
commented
2 years ago Hi @MidsummerNight! The eval calls in the Makefile specify also the fuzzer's dependency.
Take the following for instance:
$(eval $(call fuzzer,010-clb-lutinit,005-tilegrid,all))This runs the 010-clb-lutinit which depends on the 005-tilegrid one, so, in case the tilegrid one was not called, than it would get built first, prior to running the 010-clb-lutinit.
In general, the 072 to 075 are used to get general information on the device, for instance 074 dumps all the information regarding tiles and their connections, and generates the tileconn.json file.
In theory you could directly run make -j$(nproc) from the fuzzer directory and it would build all of them in the correct order.
MidsummerNight
commented
2 years ago Update: 072, 073 completed successfully, while 074 failed as expected I'll try to figure out what wires to ignore tomorrow. I expect all fuzzers save for 051 should pass since I'm basically trying to replicate @MaksRawski's results with the latest Project X-Ray code at the moment.
Thanks for the heads-up @acomodi ! Is 071 also used for general info on the device like 072 - 075 ? Am I right to think that 000 - 007 are preparation steps, 010 - 066 test different FPGA logic elements, and 071 - 075 extracts general info on the device? And where can I learn more about each fuzzer? Unfortunately I find the official documentation to be very vague for me.
MidsummerNight
commented
2 years ago Update: 074 passed by following the guide, along with 075, 010~034. 034 requires 071 to be finished first. Will take a break with prjxray this weekend and continue next week.
MidsummerNight
commented
2 years ago Update : 035~060 all completed successfully (as in having generated the run.ok file), including 051 that had issues in #1534. 061 failed right after running make -f prjxray/utils/top_generate.mk:
+ make -f prjxray/utils/top_generate.mk
make[2]: Entering directory 'prjxray/fuzzers/061-pcie-conf/build_xc7s50fgga484-1/specimen_001'
if [ -f prjxray/fuzzers/061-pcie-conf/top.py ] ; then python3 prjxray/fuzzers/061-pcie-conf/top.py >top.v; fi
Traceback (most recent call last):
File "prjxray/fuzzers/061-pcie-conf/top.py", line 94, in <module>
main()
File "prjxray/fuzzers/061-pcie-conf/top.py", line 46, in main
site_name, site_type = gen_sites()
TypeError: 'NoneType' object is not iterable
prjxray/utils/top_generate.mk:8: recipe for target 'top.v.ok' failed
make[2]: * [top.v.ok] Error 1
make[2]: Leaving directory 'prjxray/fuzzers/061-pcie-conf/build_xc7s50fgga484-1/specimen_001'
Makefile:23: recipe for target 'build_xc7s50fgga484-1/specimen_001/OK' failed
make1: [build_xc7s50fgga484-1/specimen_001/OK] Error 2
make1: Leaving directory 'prjxray/fuzzers/061-pcie-conf'
Makefile:27: recipe for target 'run' failed
make: ** [run] Error 2
make: Leaving directory 'prjxray/fuzzers/061-pcie-conf'Is this because Spartan-7 chips lack PCIe to start with?
Also can anyone help me check if my spartan7_50.sh is correct? I don't know how should I set the values starting from XRAY_ROI_FRAMES so I just copied @MaksRawski's script. I don't really understand the meaning each variable stands for, and don't know how to choose their values as a result.
Another question is I see artix7 has a database/artix7/harness/arty-a7/swbut/design.json file that's used by symbiflow-arch-defs to generate channels.db. Is this file also generated from a fuzzer (I don't see one), and how should I create one?
bunnie
commented
2 years ago I can't say if it's correct or not, but this is the script I used for the extraction run I did and it produced valid enough results, but could not get 100% coverage:
export XRAY_DATABASE="spartan7"
export XRAY_PART="xc7s50csga324-1il"
export XRAY_ROI_FRAMES="0x00000000:0xffffffff"
# All CLB's in part, all BRAM's in part, all DSP's in part.
# tcl queries IOB => don't bother adding
export XRAY_ROI_TILEGRID="SLICE_X0Y0:SLICE_X65Y99 SLICE_X0Y100:SLICE_X57Y149 RAMB18_X0Y0:RAMB18_X1Y59 RAMB36_X0Y0:RAMB36_X1Y29 RAMB18_X2Y0:RAMB18_X2Y39 RAMB36_X2Y0:RAMB36_X2Y19 DSP48_X0Y0:DSP48_X1Y59"
export XRAY_EXCLUDE_ROI_TILEGRID=""
# These settings must remain in sync
export XRAY_ROI="SLICE_X0Y100:SLICE_X35Y149 RAMB18_X0Y40:RAMB18_X0Y59 RAMB36_X0Y20:RAMB36_X0Y29 DSP48_X0Y40:DSP48_X0Y59 IOB_X0Y0:IOB_X0Y49"
# Most of CMT X0Y2.
export XRAY_ROI_GRID_X1="10"
export XRAY_ROI_GRID_X2="58"
# Include VBRK / VTERM
export XRAY_ROI_GRID_Y1="0"
export XRAY_ROI_GRID_Y2="49"
export XRAY_PIN_06="R12"
export XRAY_PIN_05="V12"
export XRAY_PIN_04="U11"
export XRAY_PIN_03="T11"
export XRAY_PIN_02="R11"
export XRAY_PIN_01="T13"
export XRAY_PIN_00="T12"
source $(dirname ${BASH_SOURCE[0]})/../utils/environment.shI did not run 061 in my setup, because the Spartan7 does not have PCIe.
bunnie
commented
2 years ago (fwiw I also had no idea how to set the ROI frames, I think I just did some wild flailing around guesses until things seemed to work)
MidsummerNight
commented
2 years ago @bunnie Could the reason be you're using xc7s50csga324-1il instead of xc7s50fgga484-1? Step 2 suggests "choosing a package that is fully bonded".
And how did you check how valid your results are? My plan is to (assuming all fuzzers run well) run the SymbiFlow Architecture Definitions flow with all the xc7s50 data generated as its input, produce the chip's arch.timing.xml and Resource Routing Graph in arch-def's flow, then use them to generate bitstreams of a few designs (like some simple counters) and load the bitstream to a board to see if my fuzzing results are correct. The fundemental problem with this approach is it's such a lengthy process that it'll be hard to pin-point what step went wrong if the bitstream doesn't work as intended, while the immediate blockade is that the arch-defs flow requires that design.json file I mentioned above which seems to be board-dependent and I don't know how to create it.
@acomodi @mithro @kgugala Can you help explain how to correctly setup the shell script and how to create the design.json file? Thanks!
By the way, just for reference my prjxray folder already takes up 40+GB so far testing xc7s50fgga484-1 alone. It seems most of these data will become useless once X-Ray's tools are done analyzing them, in fact fhe final results written to prjxray/database are just 50MB or so, but anyone trying to run these tests in a VM should definitely watch out for their disk space.
bunnie
commented
2 years ago I'm specifically using an xcs50csga324-1il part in a design, and I needed the Prjxray results for a very specific need, which was to create a bitstream patcher for me to insert root encryption keys into the bitstream so that the entire thing can can be protected by the 256-bit AES eFuse key without needing the assistance of any external provisioning computer (for more details you can see the script that generates the patch locations from the Prjxray output).
So far I have not seen any failures in the results, and fwiw, the locations on the S7 basically line up with those or the Artix for this particular ROM implementation, so I have fairly high confidence in the outcome.
My prjxray directory ate up 83G total, before I had sufficient coverage to extract the bits I needed. Like I said, the run didn't complete 100%, but it got far enough to map out the bits I needed, so I stopped running it. Like you, I had to migrate this off of a VM that I was running this in and move it to a dedicated box that we were using for CI of our designs, and it pretty much munched away for a couple days before getting the results we needed.
mithro
commented
2 years ago It wouldn't be unusual for the prjxray to end up >100gb disk usage.
acomodi
commented
2 years ago @MidsummerNight With the work on the FPGA Interchange proceeding, I think that you might try that flow instead of going through SymbiFlow arch defs, which, as you say, might add too much overhead for your needs.
You may look into https://github.com/SymbiFlow/fpga-interchange-tests, which is used to test the interchange format, for now using nextpnr-fpga_interchange and in the future also VPR (which support for the interchange is currently under development).
With the Interchange tests you would not need to worry as much for the RR graph and arch.timing.xml data generation as it is already present in the interchange device file provided by RapidWright.
I think you just might need to provide the spartan7 device definition, similarly to what happens for the artix7, the board you want to test with, and enable the tests for that board (provided you put a valid XDC in the test directory with the same name as the board).
The flow should take you down to the FASM generation from which you can potentially get a bitstream using the database containing the spartan7 data as well
GitHub
Repository to run extensive tests on the FPGA interchange format - GitHub - SymbiFlow/fpga-interchange-tests: Repository to run extensive tests on the FPGA interchange format
MidsummerNight
commented
2 years ago @bunnie I see. My goal is to allow SymbiFlow to be able to generate bitstreams targeting Spartan-7 from user HDL designs, preferrably using the Yosys-VPR-FASM-Bitstream flow (which would require me to get the RR graph and arch.timing.xml) since it already supports a few Artix-7 parts, while nextpnr says their support for Project X-Ray is still under development.
@acomodi Thanks! I'll look into this Interchange thing.
MidsummerNight
commented
2 years ago Update: 062, 063 failed, 065, 065b succeeded, 066 failed, 076, 100 and 101 succeeded. Things might slow down for a while as I figure out what to do with all the generated data.
acomodi
commented
2 years ago @MidsummerNight I believe that it is great progress. I suggest that you might soon put together a PR and let CI run all the fuzzers, so we have even more data on what is failing/succeeding and possibly help finding out what is happening.
For what I see, all the failing fuzzers are not "critical" as they are for documenting features for advanced hard blocks, which are not relevant for initial simple designs.
MidsummerNight
commented
2 years ago Update: Made a pull request. Hopefully this gets merged soon!
mithro
commented
2 years ago A new prjxray-db was published with this initial spartan7 support @ https://github.com/SymbiFlow/prjxray-db/tree/master/spartan7
The HTMLified version can be found at https://symbiflow.github.io/prjxray-db/#database-for-spartan7 with the HTML rendered tile graph found at spartan7.
Project X-Ray Database: XC7 Series. Contribute to SymbiFlow/prjxray-db development by creating an account on GitHub.
MidsummerNight
commented
2 years ago @mithro @acomodi Great! Now what should I do to improve Spartan-7 support? I see 2 tasks in my sight:
(Long term) From the htmlgen report it seems that there's lots of information about the chip yet to be documented, so how to I work on that front? Should I try to improve the fuzzers right away (which is quite challenging for me and can potentially impact the fuzzing of other families), or do I start with adjusting parameters in settings/spartan7.sh (also challenging but at least this won't affect other chips)? Any tips on what to do and how to do it?
(Relatively short term) Now we have initial prjxray-db info on spartan7, I think its time to put it into the arch-defs process and eventually build a Spartan-7 test for symbiflow-examples. Problem is, I can't understand what arch-defs' CMakeList is doing (I know the syntax, but I don't know where all the targets, dependencies etc. are located, what do they contain and how do they work together). How can I correctly add a test for spartan7 in arch-defs (I do have xc7s50 hardware and proper RTL tests files, so I need help generating the correct Makefiles via CMake)?
acomodi
commented
2 years ago @MidsummerNight I believe that the "Relatively short term" task can be split in two:
2a. (Even shorter term) Try to get to place and route a design using the Interchange format which currently is supported by nextpnr and support to VPR is being added as well. You may make use of this repository which has a much simpler CMake structure. You may need to add some definition for the Spartan7 device, board and add it to a test.
2b. (Getting Spartan7 into symbiflow-examples) SymbiFlow-examples still relies on the arch-defs generated architectures. I believe you may initially just copy what's there for the artix7 into a spartan7 directory, omitting all devices apart from the 50t which will need to be adapted for the Spartan7, and finally add and enable the Spartan7 to the CMakeLists.txt. Ater that you may just add a simple counter test to verify everything is working properly (e.g. https://github.com/SymbiFlow/symbiflow-arch-defs/tree/master/xc/xc7/tests/counter).
At a first glance, the Spartan7 device seems pretty much similar to an Artix7 one, therefore it may just work out of the box. One difference I saw is for instance the absence of GTP hard blocks, which you may omit in the definition of the device.
FOSS architecture definitions of FPGA hardware useful for doing PnR device generation. - symbiflow-arch-defs/xc/xc7/tests/counter at master · SymbiFlow/symbiflow-arch-defs
MidsummerNight
commented
2 years ago @acomodi Okay I'll look into those. But what about the long term task? How can I make xc7s50's bitstream documentation more complete, or will it simply become more complete with the CI automatically running the fuzzers over and over again (which doesn't seem to be the case, as most of SymbiFlow/prjxray-db's data is quite old)?
acomodi
commented
2 years ago In general, the fuzzers are pretty stable at this point and they cannot discover missing bits. It is not a matter of computation time, but rather of fuzzer completeness.
The usual strategy to understand how a db is complete is to increasingly test features, and understand how many missing bits are there.
To do so, you can start with simple designs, use Vivado to place and route them, and than use the prjxray bit2fasm tool with the verbosity option enabled (if I recall correctly) and this will give you the amount of bits missing from the DB.
For instance, this procedure is performed in the minitests.
Once you get some tests that do have some missing bits, the investigation part begins to understand what those correspond to and what fuzzer needs an update or to be written from scratch.
MidsummerNight
commented
2 years ago So improve the fuzzers it is then. Thanks for the heads up!
MidsummerNight
commented
2 years ago I'm closing this issue as spartan7 has been successfully added to prjxray-db, which was the goal. Will open new issues or PRs when I try to further improve upon X-Ray or have other topics in mind.
KKtiandao
commented
2 years ago You may look into https://github.com/SymbiFlow/fpga-interchange-tests, which is used to test the interchange format, for now using
nextpnr-fpga_interchangeand in the future also VPR (which support for the interchange is currently under development).With the Interchange tests you would not need to worry as much for the RR graph and arch.timing.xml data generation as it is already present in the interchange device file provided by RapidWright. @acomodi I have some questions on fpga-interchange project:
- How does Interchange test validate database provided by prjxray?
- Does VPR-Interchange flow generate the rrg and arch.timing.xml basing database provided by RapidWright? VPR-Interchange is a branch of fpga-interchange or an independent open-source project?
- How does RapidWright generate the database? I've seen RapidWright is in xilinx org, does this mean it provides an offical database?
acomodi
commented
2 years ago
- How does Interchange test validate database provided by prjxray?
@KKtiandao The interchange tests use the interchange python library that contains a FASM generator to output FASM out of the physical and logical netlist that are produced by the synthesis and place&route flow. The FASM file will than be used by the prjxray-tools to generate the bitstream.
- Does VPR-Interchange flow generate the rrg and arch.timing.xml basing database provided by RapidWright? VPR-Interchange is a branch of fpga-interchange or an independent open-source project?
The interchange support in VPR is currently being worked on. The interchange device data can be directly read into the internal VPR data structures, as well as the RR graph (the RR graph support is still in a PR at the moment).
The idea of the Interchange is that it provides a common structure and definition on how to define an architecture and how to represent the synthesized and placed&routed netlists, so that different tools such as RapidWright, VPR or nextpnr can read and write the same data "interchangeably".
- How does RapidWright generate the database? I've seen RapidWright is in xilinx org, does this mean it provides an offical database?
Yes, RapidWright outputs the interchange device data starting from the devices from the Xilinx database, which actually can also be accessible from Vivado.
KKtiandao
commented
2 years ago @acomodi Got it, Thanks!
Hi, I'm trying to add support for xc7s50t to Project X-Ray. Naturally I refered to the Guide to adding a new device to an existing family and tried to pick up where #1534 left. However it seems that the Project X-Ray toolchain has changed quite a bit since #1534 and the guide is about how to add devices to existing families instead of adding new devices of new families, so I'm currently stuck at where I'm supposed to
make db-prepare-artix7. For context, I'm on Ubuntu 20.04.3 with Vivado 2017.2 installed and successfully ran Step 9 (the010-clb-initfuzzer) of the Getting Started section of Project X-Ray's documentations (ran./download-latest-db.shin Step 8) forartix7. A minor problem occured when runningmake envof Step 5 (see my comments on #1521 ) but that didn't prevent me from proceeding.Here's what I did after completing
010-clb-initforartix7:prjxray/settings. The content was mostly copied from/prjxray/settings/spartan7.shand/prjxray/settings/spartan7_50.shin thespartan7_50branch of @litghost'sprjxrayfork./prjxray/setting/spartan7.spartan7as a new argument to update_parts.py and update_resources.py in/prjxray/utils.Then in a new terminal, following the instructions in Guide to adding a new device to an existing family, I ran
make db-prepare-spartan7and got the following output:Which already looks wrong, compared to what
make db-prepare-artix7gave me:Then
source settings/spartan7.shreturned this:Because
prjxray/database/spartan7/mapping/parts.yamlcontains only{}and nothing else.Did I miss something? How should I solve this? Thanks!