Random failures across different FPGA layouts

[fritzbauer]

DISCLAIMER: This probably does not require a litex fix, but I hope it helps others with similar problems and hope to get some hints to resources where I can learn about troubleshooting of such issues.

Issue

I experience various issues for some of my synthesized layouts, which can be "resolved" by changing anything unrelated and minor (e.g. (un)connect an LED output pin) in the overall design and run synthesis again.

There may be other issues which go unnoticed, but these are the symptoms I have experienced repeatedly:

• The litex bios cmdline via UART deletes typed characters. E.g. when attempting to type "serialboot" at some point it will just delete one to all characters, so it is hard to even enter the whole command.
  • When attempting to serialboot in this state it is common that it fails uploading with the message litex Got unexpected response from device 'b'E''
• The litex bios memtest shows errors
• The serialboot hangs after "Liftoff"

The synthesis itself seems to be deterministic enough that the issue is not fixed by just building again. However, when doing a minor change, it may be that none of these symptoms occur. (...or other of these symptoms occur...as you can imagine it is tedious to figure out which changes will eventually result in a working bitstream)

Environment

I am using the qmtech_5cefa2 board which has an Intel Cyclone V and Quartus Prime Design Software Version 21.1.1 Build 850 06/23/2022 SJ Lite Edition.

Confession

I do not use the standalone litex core, instead the generated verilog is included in an amaranth project (https://github.com/hansfbaier/adat-usb2-audio-interface). This project outputs Timing requirements not met during synthesis, however the audio interface part is working just fine, including fast USB2.0 communication with the PC via ULPI, the ADAT sockets and two I2S outputs to DACs. Therefore, I am wondering why the super-robust-and-slow-UART communication is showing these issues, while the much-faster-USB is working flawlessly.

Did anybody else experience such issues before? Do you have a hint, where I can learn about timing issues and the respective troubleshooting workflows?

[tcal-x]

Hi @fritzbauer , I do occasionally see such issues, for example https://github.com/google/CFU-Playground/issues/657 .

The first thing I do is check that the implementation actually met the timing constraints. But that is not always the cause of the problems, and I think there are some unresolved mysteries in my project as well.

If you're using litex_term for serial boot, you can try adding the --safe option, which will be very slow, but sometimes does allow the firmware to be loaded.

[fritzbauer]

Thank you. I tried the --safe option for serialboot, but it did not help for broken serial connection. It reminded me of https://github.com/enjoy-digital/litex/issues/773.

The first thing I do is check that the implementation actually met the timing constraints.

Yeah, that's what I am planning to do next, however I do not yet know how. So if there are any good resources to learn this timing stuff, I am thankful for links.

[tcal-x]

Yeah, that's what I am planning to do next, however I do not yet know how. So if there are any good resources to learn this timing stuff, I am thankful for links.

It would be useful if you share which FPGA device and board you're using, and also which toolchain (e.g. Vivado, yosys/nextpnr, etc.).

[fritzbauer]

I am using the qmtech_5cefa2 board which has an Intel Cyclone V and Quartus Prime Design Software Version 21.1.1 Build 850 06/23/2022 SJ Lite Edition.

[enjoy-digital]

@fritzbauer: Can you also try to build the LiteX-Boards target and see if you have the same behavior? If not, I would recommend making sure the timing consraints generated by LiteX are correctly integrated/interpreted in your build.

[fritzbauer]

I did not experience these issues for the plain LiteX target these days. However, I did observe it half a year ago for the LiteX target, when fiddling with sdcard boot.

I would recommend making sure the timing constraints generated by LiteX are correctly integrated/interpreted in your build.

Yes, I am about to try this, but wanted to ask how to do it by means of this issue.