jbush001 / NyuziProcessor

GPGPU microprocessor architecture
Apache License 2.0
1.99k stars 351 forks source link

Synthesized design doesn't boot on FPGA #183

Closed yuma1908 closed 4 years ago

yuma1908 commented 5 years ago

Hello, I am new to Nyuzi and firstly tried FPGA tutorial of “blinky” but this trial failed.

I used Ubuntus16.04.6 LTS and Quartus Prime 18.1 Lite.

The Error message is “Ping target…...….....…...target is not responding” and hence, I guess it fails in "ping_target" function. I traced serial_boot.c with inserting some “printf” instruction and found the execution failed with “select” instruction at line 97.

ready_fds was 0 and thus the read_serial_byte function returned “false”.

From this result, I doubt the USB to serial cable.

I used the "USB2.0 to RS232 DB 9Pin Flat Serial Cable (FTDI-FT232RL)" made by Cable Creation (Product Number is CD0619).

When I first inserted this cable, baud rate was 9600, which was confirmed by “stty -a < /dev/ttyUSB0”, and then I changed from 921600 to 9600 in logic_analyzer.sv, serial_boo.c, and boot.c.

After this operation failed, I changed baud rate to 921600 with “stty -F /dev/ttyUSB0 921600” and also changed relating .sv and .c files, but cannot pass “ping_target” (also fails in "select").

Could you give me some advice or guidance on this issue? Or is there alternative ways for initializing SDRAM memory using some Quartus Prime functions?

In my understandings, ./run_fpga initializes the SDRAM (instruction memory) on de2-115 board with .hex files, isn't it?

Many thanks,


Note that before trying ./run_fpga, I tried mapping, fitter, and program as follows.

The mapping and fitter using “make synthesis” seems to work well. (# of logic elements and STA results are similar to the README.md file)

The programming using “make program” may work well (at least from the log of Quartus Prime, no error/warning messages are found.)

On the other hand, after "make program", the Green LED 0 does not blink even when I pushed the “key 0” button for reset, which is different behavior with README.md in hardware/fpga/de2-115.


jbush001 commented 5 years ago

Thanks for the detailed notes. A few initial responses:

  1. read_serial_byte returning false indicates it timed out. This suggests the USB cable is working (otherwise it would have errored out earlier when trying to open the device), but that the board did not respond.
  2. You should not need to do an stty to change the baud rate. The serial_boot program will do that as part of its initialization. I'm a bit confused when you say you changed the baud rate in boot.c and serial_boot.c: the default rate should already be 921600. What did you change it to? (By the way, logic_analyzer.sv is unused in this case and doesn't need to be modified)
  3. ./run_fpga loads the HEX file onto the board, yes (it does this by communicating with the boot loader program). There are currently not other ways to load programs onto the board.
  4. The fact that the LED is not blinking after doing make program probably explains why you're getting the error from serial_boot. This suggests the bootloader is not working correctly (which would explain the error in number 1)

I'm afraid I don't have a good theory immediately why this would be failing. It's possible I broke something since the last time I tested on FPGA. I will look into this and add some more questions/suggestions.

yuma1908 commented 5 years ago

Thank you for quick and variable responses!

As for the first and third responses. I get a point. Since I'm not familiar with using FPGA and serial cables, I had doubted the USB cable failure. Thank you for the correction. --- Note ---

Actually, I have two de2-115 boards and both of them generates the same issue, i.e. timeout when ./run_fpga at ping_target, and I guess the FPGA itself is fine.

Also, I first used the PL2303 type USB to serial cables and got the same issues (Since the README.md notes the risk of using PL2303 chipset, I changed to use FTDI chipset one)


Sorry for the confusion about the second part. I changed the rate to 9600 (#define DEFAULT_UART_BAUD 9600).

I referred the lines 31 - 33 in "hardware/fpga/de2-115/README.md".

I note that I do ./run_fpga with following conditions;

(1) no stty for /dev/ttyUSB0 && #define DEFAULT_UART_BAUD 921600

(2) no stty for /dev/ttyUSB0 && #define DEFAULT_UART_BAUD 9600

"stty -a < /dev/ttyUSB0" commnds output "speed is 9600", and hence I set 9600.

(3) stty -F /dev/USB0 921600 && #define DEFAULT_UART_BAUD 921600

Anyway, I understand that I should not need to change the baud rate using "stty".

Thank you for the suggestion about the fourth point. Firstly, I will check the issue #1. Then, I will checkout a little older version from the git and will try run_fpga.

Many thanks.

yuma1908 commented 5 years ago

I checked out the 9d205e30c147dbf1736082bb33ba97ca2a843c17 (1-year before version) and tried the FPGA blinking tutorial. After "make program" is done, the LED turns to repeat "on" and "off", which is a better point. However, when I running the ./run_fpga, the load_memory fails.

I did not change the baud rate, i.e., did not use "stty" and also did not modify "DEFAULT_UART_BAUD".

..... Error log (The ping passes but memory_load fails.) .... ping target...... Program is 8192 bytes _loading [ ] (0%) 00000000 Did not get ack for load memory, got c8 instead 963 BAD_COMMAND bytes seen, last was c8 byte read: c8 Ping return seen. . . . Cannot fix connection, no ping from board recieved. Try resetting the board (KEY0) and rerunning. ......

SDRAM control or memory access may fail due to some issues. I will checkout another version and retry the FPGA blinky tutorial.

Many thanks.

jbush001 commented 5 years ago

Interesting. The byte c8 that is read means 'bad command', which means the bootloader is working and able to send characters correctly to the host, but somehow it doesn't recognize characters sent from the host. I don't have a theory why it would fail in that way.

Sorry you're having issues, and thank you for helping debug! I'm working on getting my board up again, but, unfortunately Quartus jtagd is being finicky again for me (I'm guessing an OS update broke something), so I'll need to get that working again.

yuma1908 commented 5 years ago

Thank you for comments!

I further check-outed following three versions and retry: (1) 2017 Apr. 22 (c9e96670632560fdd1b989aa2a7137fd8bdea265) (2) 2016 Apr. 13 (e329aa81a146c901f9a6de63b75157db17b061b2) (3) 2015 Apr. 26 (092441189c94a5bf88ebdb117e4b57f03307551e)

As for (1) and (2), I got the same error (ping passes but memory read failed.)

The read byte is c8 and in (1) and 00 in (2).

On the other hand, in (3), I confirmed ./runit.sh blinks LED. Actually, in (3), SDRAM seems to be initialized with USB blaster, not RS232. From this result, now I re-doubt some serial-cable issue. (Because SDRAM initialization and address-mapping seem to work well.)

Another candidate may be some version compatibility problem such as Quartus or Ubuntu or other packages.

In the next step, I will check-out the version (4) 2015 Apr. 29 (c81e05cbfae0bb1189d5ccf611d71a7a9c9bfe0c), and will confirm whether this relatively initial version of using RS232 works well.

If this version does not work well, not Hardware issue but RS232 or some compatibility issues may cause the problem.

By the way, today I ordered your recommend RS232 cable which is described in README.md in de2-115 setup for setting identical conditions and thus reducing the possible error reasons. (If I get errors using this ordered cable, I will move on to some version checking for Quartus and so on)

Many thanks!

jbush001 commented 5 years ago

2015 is pretty old. The problem with running builds that are that old is that I've made changes to the instruction set periodically, so you need to use the right version of the compiler (before I checked in the compiler as a subproject, there wasn't an explicit link either). Anyway, I've have had it working on my board since then. For example, I was debugging issues in hardware last year, see issue #140).

Yes, as you noticed, in the 2015 build, you could load a small chunk of code into internal SRAM using the JTAG blaster and a tool my friend wrote called 'jload', and from there put code into SDRAM over serial by manually loading the bootloader. The whole memory layout has changed since then. Unless it is of historical interest, I wouldn't recommend spending too much time on those old builds, as they were a bit clunkier and harder to use.

Anyway, it seems like something is going on with the serial. Another person had some serial problems as well. https://gitter.im/jbush001/NyuziProcessor?at=570da33f3ddb73ba105b815b Unfortunately, we never got to the bottom of it, and I couldn't reproduce on my setup.

yuma1908 commented 5 years ago

Thanks for variable suggestions. OK, I will wait for the arrival of the ordered RS232 cable and will retry the relatively newer version, e.g., 2018 Apr. Unfortunately, the arrival will one week later, and hence, I will report the results after that.

Also, thank you for the information about relating issue #140 and the URL. I will check both.

Many thanks.

yuma1908 commented 5 years ago

Hello,

I got the RS232 serial cable which you recommended in README.md. Then, I tried FPGA blinky trial and mandelbrot trial. In both tutorials, I observed the expected behavior !!!

In other words, the previous unexpected behavior is probably due to RS232 cable issue.

Thank you very much for your kind helps and variable suggestions !

Note: I used a little older commit, i.e. Aug 12, 2017 (283518b296c9f96842ebda05bdb2b80671c4b845)

In the latest commit, the ping_target did not pass.

The reason why I used two-years old verision is that I will also want to do VCS trial.

(Actually, I confirmed correct operations for mandelbrot and hash program in RTL simulation with VCS in this commit version, and now try to synthesize RTL netlist using Design Compiler and following Logic simulation with VCS.)

Many thanks.

jbush001 commented 5 years ago

Thank you for the update, I'm glad to hear it's working for you now! I would still like to get to the bottom of why the latest version isn't working and will continue to investigate that.

schuler1 commented 5 years ago

I am having the same issue described above. After pressing the reset button, the green LED does not blink or turn on. In a prior version of nyuzi, things used to work with this USB to serial cable: http://www.usconverters.com/usb-rs232-adapter-xs8801

This cable has a green LED on the adapter. When I run_fpga, I see the green LED blinking. A similar red LED blinks on the board. But the run_fpga fails with target is not responding.

Simpler checks on the adapter give the appearance of things being OK:

`ubuntu:64% lsusb Bus 001 Device 002: ID 0e0f:000b VMware, Inc. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 008: ID 09fb:6001 Altera Blaster Bus 002 Device 005: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC Bus 002 Device 004: ID 0e0f:0008 VMware, Inc. Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

ubuntu:65% dmesg | grep FTDI [ 6.527955] usb 2-2.2: Manufacturer: FTDI [ 11.978579] usbserial: USB Serial support registered for FTDI USB Serial Device [ 11.978641] ftdi_sio 2-2.2:1.0: FTDI USB Serial Device converter detected [ 12.217511] usb 2-2.2: FTDI USB Serial Device converter now attached to ttyUSB0

ubuntu:67% uname -a Linux ubuntu 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux`

Given that my cable is pretty good (I think) I am a bit hesitant about concluding the cable is the issue.

I have modified boot.c to add the red LED indication in addition to the blinking green LED. The red LED doesn't come up. So I don't think boot.c is getting very far...

`int main() { // Turn on red LED to indicate bootloader is waiting REGISTERS[REG_RED_LED] = 0x1;

// Initialize UART speed
REGISTERS[REG_UART_DIVISOR] = (CLOCK_RATE / DEFAULT_UART_BAUD) - 1;

for (;;)
{`
jbush001 commented 5 years ago

Yes, I believe there are two issues in this report:

  1. The latest version of the RTL does not work correctly on FPGA.
  2. The serial cable was not working properly in the reporters configuration using an older (working) version of the hardware.

The last few comments were related to 2, but 1 is still an issue. I believe I must have broken something in hardware with one of the newer changes. It does work still in simulation, so it is presumably either:

  1. A change is in a piece of logic that is only used during synthesis. It could be a change directly in de2_115_top.sv (although there have only been a few ostensibly minor changes in there since the last time I tested with hardware). It could also be a change in the nyuzi core broke a contract that de2_115_top was expecting.
  2. A synthesis/simulation mismatch has been introduced somewhere in the core RTL.

I tried to test this on my board, but, unfortunately, jtagd has stopped working again (I'm guessing an OS update broke it), and I haven't had time to debug it.

achavez017 commented 4 years ago

I too was having issues with the byte C8 error, but was able to get around it by lowering the baud rate to 115200.

Regarding loading the project onto the FPGA, the most recent version I was able to get to work properly was https://github.com/jbush001/NyuziProcessor/commit/2c00dcdaade6b2b813aed2ae7064fc2ade2e0f7d from December 1, 2018. This version, https://github.com/jbush001/NyuziProcessor/commit/eb2f720d53ea4ff3450e78a30e1cdb53cd3357bb, from December 2, 2018 no longer blinks LEDG0, and programs can no longer be uploaded through the serial port.

jbush001 commented 4 years ago

Thank you for isolating the build where it broke, that is super helpful! It looks like this happened right after I rebased the toolchain from LLVM upstream. This suggests perhaps it is no longer generating the boot ROM image correctly. I will investigate that.

jbush001 commented 4 years ago

Sure enough, I modified start.S to add the following (which is 0xdeadbeef, byteswapped)

_start: .long 0xefbeadde

I then compiled and looked at boot.hex. I would expect this to be the first word in the file, but it is at offset 220. I'm using the linker script boot.ld and the --oformat,binary flag to produce a raw binary, but it looks like something is not working as I expect.

jbush001 commented 4 years ago

I then modified boot.c to add constant data:

const char *stuff = "congratulations, you found the secret message";

When I look at the hexdump, I now see that in the beginning section:

68e3feff
78e4feff
38e4feff
78e4feff
6ce4feff
78e4feff
04e4feff
636f6e67   cong <------
72617475   ratu
6c617469   lati
6f6e732c   ons 
20796f75
20666f75

The issue is that the linker has put the read only data section before the code. This is what it does for normal ELF binaries, but it obviously breaks a raw binary. I should see if this is a problem with the latest toolchain, and perhaps file a bug against LLD if that is the case. In the interim, I can work around this by using elf2hex to generate the binary instead of the linker flags. The only disadvantage of that is that it has the ELF header still, which wastes space.

jbush001 commented 4 years ago

If you have a chance, you could try with top-of-tree, which has a potential fix. I have not had a chance to test it yet.

achavez017 commented 4 years ago

No luck, unfortunately. It now fails during synthesis. It returns the following errors:

Error (10170): Verilog HDL syntax error at boot.hex(1) near text: . Check for and fix any syntax errors that appear immediately before or at the specified keyword. The Intel FPGA Knowledge Database contains many articles with specific details on how to resolve this error. Visit the Knowledge Database at https://www.altera.com/support/support-resources/knowledge-base/search.html and search for this specific error message number. File: /home/abraham/Nyuzi/NyuziProcessor/software/bootrom/boot.hex Line: 1 Error (12152): Can't elaborate user hierarchy "axi_rom:boot_rom" File: /home/abraham/Nyuzi/NyuziProcessor/hardware/fpga/de2-115/de2_115_top.sv Line: 142 Error: Quartus Prime Analysis & Synthesis was unsuccessful. 2 errors, 24 warnings Error: Peak virtual memory: 1237 megabytes Error: Processing ended: Thu Oct 17 22:37:15 2019 Error: Elapsed time: 00:00:26 Error: Total CPU time (on all processors): 00:00:42

jbush001 commented 4 years ago

Oops, yeah, I see what I did wrong. The elf2hex tool emits address references, which the Altera tools don't support.

jbush001 commented 4 years ago

I believe the last change should address the issue, but have not had a chance to test it yet.

achavez017 commented 4 years ago

Yes. It's working again. Thank you.

jbush001 commented 4 years ago

Phew, thank you for testing!