search

pavel-demin / red-pitaya-notes

Notes on the Red Pitaya Open Source Instrument
http://pavel-demin.github.io/red-pitaya-notes/
MIT License
339 stars 211 forks source link

Question about interfacing with the FPGA using mmap(), and /dev/mem protection #1016

Closed bgergely0 closed 3 years ago

bgergely0 commented 3 years ago

Description of the setup:

Description of the problem:

Im trying to use a custom FPGA image. I communicates with the Arm cores using the AXI-GPIO IP cores. I have 2 memory locations: 00x42000000, and 0x41200000. When my client tries to mmap these locations, the Linux dies (without any error messages). I tried using busybox devmem on the memory locations as well, with the same results. (Which is not surprising, since busybox also uses mmap in the background) My exact commands:

if((fd = open(name, O_RDWR)) < 0)
  {
    perror("open");
    return 1;
  }
cfg = mmap(NULL, sysconf(_SC_PAGESIZE), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x42000000);
ptr = mmap(NULL, sysconf(_SC_PAGESIZE), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x41200000);

When using busybox devmem on smaller addresses, for example leaving one 0, so busybox devmem 0x4200000, it works.

My question is: Are these memory locations protected by the Linux kernel? If so, how can I configure the Linux to allow these locations? Or how can I find other locations to use?

Steps to reproduce the problem:

  1. Upload the provided Debian image to the board
  2. Try displaying memory locations with busybox devmem
pavel-demin commented 3 years ago

I do not think that these memory locations are protected. I think that the problem is on the FPGA side.

In the steps to reproduce the problem, you do not mention programming the FPGA with your bitstream file. If the FPGA is not programmed with a configuration that recognizes addresses 0x42000000 and 0x41200000, then it is normal for the CPU to freeze.

If the FPGA is programmed and your FPGA configuration uses the FCLK0 clock, then the FCLK0 clock may not be running when your code tries to access addresses 0x42000000 and 0x41200000. In this case, the solution is to enable the FCLK0 clock.

It is possible to enable the FCLK0 clock and set its frequency from the Linux command line using the following commands:

devcfg=/sys/devices/soc0/amba/f8007000.devcfg
echo fclk0 > $devcfg/fclk_export
echo 1 > $devcfg/fclk/fclk0/enable
echo 125000000 > $devcfg/fclk/fclk0/set_rate
bgergely0 commented 3 years ago

It turns out that the problem is indeed with the FPGA clocks, as you mentioned. I connected 1 user led to a counter running from fclk0 @ 125 MHz, one to fclk1 @ 250 MHz, and one to constant one (the counters' outputs have around 1-2 Hz frequencies). After cat system_wrapper.bit > /dev/xdevcfg only the constant one lights up.

I tried the fclk configuration you mentioned, however there is no fclk directory under $devcfg for me, and I cannot create one (Operation not permitted). I downloaded the prebuilt Debian image. This is how the directory looks for me:

root@red-pitaya:/sys/devices/soc0/amba/f8007000.devcfg# ll
total 0
-rw-r--r-- 1 root root 4096 Sep 16 11:43 aes_en_lock
-rw-r--r-- 1 root root 4096 Sep 16 11:43 dbg_lock
lrwxrwxrwx 1 root root    0 Sep 16 11:43 driver -> ../../../../bus/platform/drivers/xdevcfg
-rw-r--r-- 1 root root 4096 Sep 16 11:43 driver_override
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_aes
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_dap
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_dbg_in
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_dbg_nonin
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_sec_dbg_in
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_sec_dbg_nonin
-rw-r--r-- 1 root root 4096 Sep 16 11:43 enable_seu
-rw-r--r-- 1 root root 4096 Sep 16 11:43 fclk_export
-rw-r--r-- 1 root root 4096 Sep 16 11:43 fclk_unexport
-rw-r--r-- 1 root root 4096 Sep 16 11:43 is_partial_bitstream
-r--r--r-- 1 root root 4096 Sep 16 11:43 modalias
lrwxrwxrwx 1 root root    0 Sep 16 11:43 of_node -> ../../../../firmware/devicetree/base/amba/devcfg@f8007000
drwxr-xr-x 2 root root    0 Sep 16 11:43 power
-rw-r--r-- 1 root root 4096 Sep 16 11:43 prog_done
-rw-r--r-- 1 root root 4096 Sep 16 11:43 seu_lock
lrwxrwxrwx 1 root root    0 Sep 16 11:31 subsystem -> ../../../../bus/platform
-rw-r--r-- 1 root root 4096 Sep 16 11:31 uevent
drwxr-xr-x 3 root root    0 Sep 16 11:31 xdevcfg
root@red-pitaya:/sys/devices/soc0/amba/f8007000.devcfg# cat fclk_export 
fclk0
fclk1
fclk2
fclk3
root@red-pitaya:/sys/devices/soc0/amba/f8007000.devcfg# mkdir fclk
mkdir: cannot create directory 'fclk': Operation not permitted

I also tested the clocks from u-boot, with these results:

Zynq> clk dump
clk             frequency
    armpll          1333333320
    ddrpll          1066666656
     iopll           999999990
 cpu_6or4x           666666660
 cpu_3or2x           333333330
    cpu_2x           222222220
    cpu_1x           111111110
     ddr2x           355555552
     ddr3x           533333328
       dci            10158730
     lqspi            33333333
       smc            22222222
      pcap           199999998
      gem0           124999999
      gem1            16666667
     fclk0            50000000
     fclk1            41666666
     fclk2            41666666
     fclk3            41666666
      can0             8000000
      can1             8000000
     sdio0            99999999
     sdio1            99999999
     uart0            99999999
     uart1            99999999
      spi0           166666665
      spi1           166666665
       dma           222222220
 usb0_aper           111111110
 usb1_aper           111111110
 gem0_aper           111111110
 gem1_aper           111111110
sdio0_aper           111111110
sdio1_aper           111111110
 spi0_aper           111111110
 spi1_aper           111111110
 can0_aper           111111110
 can1_aper           111111110
 i2c0_aper           111111110
 i2c1_aper           111111110
uart0_aper           111111110
uart1_aper           111111110
 gpio_aper           111111110
lqspi_aper           111111110
  smc_aper           111111110
      swdt           111111110
   dbg_trc            66666666
   dbg_apb            66666666
Zynq>

So the clocks are running now. I used tftpboot to load the bitfile onto the Red Pitaya, then used fpga loadb to load it. After this, the leds were working.

Is there any other way to enable fclk-s and set the correct frequencies in the Linux? (Or should the default values be changed in the first stage bootloader?) (When uploading the fpga image from u-boot, the outputs were also generated on the addresses described above, so the issue is only with the clocks)

Thanks for your time!

pavel-demin commented 3 years ago

Have you tried all the commands (including echo fclk0 > $devcfg/fclk_export) from my previous comment?

I think the fclk directory is automatically created by echo fclk0 > $devcfg/fclk_export.

The clocks are configured via the device tree file. To configure the FPGA and clocks at boot you'll need to rebuild devicetree.dtb and boot.bin with the following commands:

make NAME=<your_project> devicetree.dtb
make NAME=<your_project> boot.bin

Then just copy these files to the first partition on the SD card.

bgergely0 commented 3 years ago

I didnt run echo fclk0 > $devcfg/fclk_export originally because the fclk_export already contained the fclks. But I did now, and it generated the files needed for the setup. The leds, and my fpga bitfiles seems to work now. I will try setting the default frequencies in the devicetree. Thanks for your help!

pavel-demin commented 3 years ago

because the fclk_export already contained the fclks

These are special files. The read and write operations with these files are not always related. Here is a link to some information about these files: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841795/Controlling+FCLKs+in+Linux