search

myriadrf / xtrx_linux_pcie_drv

XTRX PCI driver for linux
Apache License 2.0
16 stars 22 forks source link

Compilation errors on gcc 7.3 on arm64 4.4.132 kernel. #5

Closed gustad closed 5 years ago

gustad commented 5 years ago

I'm trying to build the driver on an arm64 4.4.132 based system using gcc 7.3, but I'm getting the compilation errors as shown below:

$uname -a
Linux rockpro64 4.4.132-1075-rockchip-ayufan-ga83beded8524 #1 SMP Thu Jul 26 08:22:22 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$ gcc --version
gcc (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ git rev-parse --short HEAD
815d890
$ make
make -C /lib/modules/4.4.132-1075-rockchip-ayufan-ga83beded8524/build M=/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git modules
make[1]: Entering directory '/usr/src/linux-headers-4.4.132-1075-rockchip-ayufan-ga83beded8524'
  CC [M]  /home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.o
/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.c: In function 'xtrxfd_mmap':
/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.c:881:24: error: implicit declaration of function 'bus_to_virt'; did you mean 'fix_to_virt'? [-Werror=implicit-function-declaration]
    (long unsigned int)(bus_to_virt(addr)),
                        ^~~~~~~~~~~
                        fix_to_virt
/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.c:882:37: warning: passing argument 1 of 'virt_to_phys' makes pointer from integer without a cast [-Wint-conversion]
error, forbidden warning: xtrx.c:882
scripts/Makefile.build:283: recipe for target '/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.o' failed
make[2]: *** [/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.o] Error 1
Makefile:1471: recipe for target '_module_/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git' failed
make[1]: *** [_module_/home/projects/sdr/xtrx/xtrx_linux_pcie_drv.git] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.4.132-1075-rockchip-ayufan-ga83beded8524'
Makefile:10: recipe for target 'modules' failed
make: *** [modules] Error 2

I've made a patch:

From a4560c255c8d8090ad34bc3a521bd3e23054ec54 Mon Sep 17 00:00:00 2001
From: Petter Gustad <petter@gustad.com>
Date: Mon, 8 Oct 2018 21:08:41 +0200
Subject: [PATCH] Fixed compilation errors on arm64/gcc 7.3

* Removed unused function.
* Replaced bus_to_virt with phys_to_virt.
---
 xtrx.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/xtrx.c b/xtrx.c
index c7e35cf..861eb02 100644
--- a/xtrx.c
+++ b/xtrx.c
@@ -663,13 +663,6 @@ static irqreturn_t xtrx_irq_legacy(int irq, void *data)
    return IRQ_HANDLED;
 }

-static irqreturn_t xtrx_msi_irq_single(int irq, void *data)
-{
-   xtrx_irq_legacy(irq, data);
-   return IRQ_HANDLED;
-}
-
-
 static struct pps_source_info xtrx_pps_info = {
    .name       = "xtrx_pps",
    .path       = "",
@@ -876,11 +869,11 @@ static int xtrxfd_mmap(struct file *filp, struct vm_area_struct *vma)
        printk(KERN_NOTICE PFX "mmap() call: VMA=%p vma->vm_pgoff=%lu addr=%p dev_addr=%lx bus_addr=%lx phys_addr=%lx\n",
            vma, vma->vm_pgoff, (vma->vm_pgoff == 0) ? xtrxdev->bar0_addr : xtrxdev->bar1_addr,
            (long unsigned int)addr,
-           (long unsigned int)(bus_to_virt(addr)),
-           (long unsigned int)(virt_to_phys(bus_to_virt(addr))) );
+           (long unsigned int)(phys_to_virt(addr)),
+           (long unsigned int)(virt_to_phys(phys_to_virt(addr))) );

        if (io_remap_pfn_range(vma, vma->vm_start,
-           virt_to_phys(bus_to_virt(addr)) >> PAGE_SHIFT,
+           virt_to_phys(phys_to_virt(addr)) >> PAGE_SHIFT,
            vma->vm_end - vma->vm_start,
            vma->vm_page_prot))
            return -EAGAIN;
-- 
2.17.1

Or the commit can be found here:

https://github.com/gustad/xtrx_linux_pcie_drv/commit/a4560c255c8d8090ad34bc3a521bd3e23054ec54

Note that I have not been able to make the driver work on arm64. But it's at least compiling, and the above compiles and is functional on x86_64, i.e. the driver loads and I can run test_xtrxll with the same result as 1158c4f.

gustad commented 5 years ago

The commit can be found here if the direct link to the commit does not work: https://github.com/gustad/xtrx_linux_pcie_drv But the patch above could also be applied by "git am"

sergforce commented 5 years ago

Which ARM hardware and platform are you using?

Can you also provide dmesg output when the driver is loaded? and when you run test_xtrx

gustad commented 5 years ago

It's "aarch64". See full uname output in my first message. It's the RockPro64

It's visible using lspci:

root@rockpro64:~# lspci 
00:00.0 PCI bridge: Rockchip Inc. RK3399 PCI Express Root Port Device 0100
01:00.0 Memory controller: Xilinx Corporation Device 7012

When I insmod I get the following in my kernel log:

[  182.829845] xtrx: loading out-of-tree module taints kernel.
[  182.833139] xtrx: Initializing 0000:01:00.0
[  182.833193] xtrx 0000:01:00.0: enabling device (0000 -> 0002)
[  182.834145] pps pps0: new PPS source xtrx_pps
[  182.835317] xtrx: buf[0]=f2200000 [virt ffffff80095c1000] => f22007ff
[  182.835349] xtrx: buf[1]=f2208000 [virt ffffff80095c9000] => f22087ff
[  182.835377] xtrx: buf[2]=f2210000 [virt ffffff80095d1000] => f22107ff
[  182.835405] xtrx: buf[3]=f2218000 [virt ffffff80095d9000] => f22187ff
[  182.835432] xtrx: buf[4]=f2220000 [virt ffffff80095e1000] => f22207ff
[  182.835459] xtrx: buf[5]=f2228000 [virt ffffff80095e9000] => f22287ff
[  182.835486] xtrx: buf[6]=f2230000 [virt ffffff80095f1000] => f22307ff
[  182.835514] xtrx: buf[7]=f2238000 [virt ffffff80095f9000] => f22387ff
[  182.835541] xtrx: buf[8]=f2240000 [virt ffffff8009601000] => f22407ff
[  182.835568] xtrx: buf[9]=f2248000 [virt ffffff8009609000] => f22487ff
[  182.835595] xtrx: buf[10]=f2250000 [virt ffffff8009611000] => f22507ff
[  182.835623] xtrx: buf[11]=f2258000 [virt ffffff8009619000] => f22587ff
[  182.835650] xtrx: buf[12]=f2260000 [virt ffffff8009621000] => f22607ff
[  182.835677] xtrx: buf[13]=f2268000 [virt ffffff8009629000] => f22687ff
[  182.835704] xtrx: buf[14]=f2270000 [virt ffffff8009631000] => f22707ff
[  182.835731] xtrx: buf[15]=f2278000 [virt ffffff8009639000] => f22787ff
[  182.835758] xtrx: buf[16]=f2280000 [virt ffffff8009641000] => f22807ff
[  182.835785] xtrx: buf[17]=f2288000 [virt ffffff8009649000] => f22887ff
[  182.835812] xtrx: buf[18]=f2290000 [virt ffffff8009651000] => f22907ff
[  182.835839] xtrx: buf[19]=f2298000 [virt ffffff8009659000] => f22987ff
[  182.835979] xtrx: buf[20]=f22a0000 [virt ffffff8009661000] => f22a07ff
[  182.836009] xtrx: buf[21]=f22a8000 [virt ffffff8009669000] => f22a87ff
[  182.836037] xtrx: buf[22]=f22b0000 [virt ffffff8009671000] => f22b07ff
[  182.836096] xtrx: buf[23]=f22b8000 [virt ffffff8009679000] => f22b87ff
[  182.836155] xtrx: buf[24]=f22c0000 [virt ffffff8009681000] => f22c07ff
[  182.836208] xtrx: buf[25]=f22c8000 [virt ffffff8009689000] => f22c87ff
[  182.836259] xtrx: buf[26]=f22d0000 [virt ffffff8009691000] => f22d07ff
[  182.836321] xtrx: buf[27]=f22d8000 [virt ffffff8009699000] => f22d87ff
[  182.836349] xtrx: buf[28]=f22e0000 [virt ffffff80096a1000] => f22e07ff
[  182.836377] xtrx: buf[29]=f22e8000 [virt ffffff80096a9000] => f22e87ff
[  182.836404] xtrx: buf[30]=f22f0000 [virt ffffff80096b1000] => f22f07ff
[  182.836431] xtrx: buf[31]=f22f8000 [virt ffffff80096b9000] => f22f87ff
[  182.836457] xtrx: Failed to allocate 0 DMA buffer
[  182.836486] xtrx 0000:01:00.0: Failed to register TX DMA buffers.

I can't run test_xtrx since I don't get any /dev/xtrx0.

However, I will make sure the board and software works on an Intel platform before I continue any Arm testing.

sergforce commented 5 years ago 
[  182.836457] xtrx: Failed to allocate 0 DMA buffer
[  182.836486] xtrx 0000:01:00.0: Failed to register TX DMA buffers.

Is an actual problem. It was able to allocate 2MB of DMA buffers for RX but wasn't for TX. It seems there's limitation of 2MB. On NVIDIA Terga this problems is resolved by adding 'coherent_pool= ' into kernel boot parameter list https://devtalk.nvidia.com/default/topic/770634/jetson-tk1/large-coherent-dma-blocks/ Unfortunatly I don't have ROCKPro64 on hands, but I'll look into docs

Can you also do lspci -vvv for XTRX?

gustad commented 5 years ago

I will do that when I move the board back to the RockPro64. But I saved the output of lspci -vv during my previous session:

root@rockpro64:~# lspci -s 01:00.0 -vv
01:00.0 Memory controller: Xilinx Corporation Device 7012
    Subsystem: Xilinx Corporation Device 0007
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 237
    Region 0: Memory at fa010000 (32-bit, non-prefetchable) [disabled] [size=4K]
    Region 1: Memory at fa000000 (32-bit, non-prefetchable) [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/16 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] Express (v2) Endpoint, MSI 01
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Device Serial Number 00-00-00-00-12-34-56-78
gustad commented 5 years ago

Does not seem like the third v provided any additional info, but here it is:

root@rockpro64:~# lspci -s 01:00.0 -vvv
01:00.0 Memory controller: Xilinx Corporation Device 7012
    Subsystem: Xilinx Corporation Device 0007
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 237
    Region 0: Memory at fa010000 (32-bit, non-prefetchable) [disabled] [size=4K]
    Region 1: Memory at fa000000 (32-bit, non-prefetchable) [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/16 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] Express (v2) Endpoint, MSI 01
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Device Serial Number 00-00-00-00-12-34-56-78
gustad commented 5 years ago

I increased coherent_pool from 1M to 2M and the driver loaded. 

./test_xtrxll 
15:27:36.857685 INFO:    Probing 'libxtrxll_pcie.so.0' low-level library
15:27:36.859466 INFO:    Probing 'libxtrxll_libusb3380.so.0' low-level library
15:27:36.860489 ERROR:   Error loading: libxtrxll_libusb3380.so.0: cannot open shared object file: No such file or directory
15:27:36.861213 REGS:    XTRX PCI_3: Read  [001a] = 04000013
15:27:36.861412 INFO:    XTRX PCI_3: XTRX Rev4 (04000013)
15:27:36.861477 INFO:    XTRX PCI_3: RX DMA STOP MIMO (BLK:0 TS:0); TX DMA STOP MIMO
15:27:36.861543 REGS:    XTRX PCI_3: Write [000d] = c0000000
15:27:36.861596 REGS:    XTRX PCI_3: Write [000d] = 42000000
15:27:36.861899 REGS:    XTRX PCI_3: Write [0001] = 00008fff
15:27:36.862007 INFO:    XTRX PCI_3: Device `/dev/xtrx0` was opened
15:27:36.862111 REGS:    XTRX PCI_3: Write [0000] = 002fffff
15:27:36.862228 DEBUG:   XTRX PCI_3: SPI[0/1] I:1
15:27:36.862324 REGS:    XTRX PCI_3: Read  [0000] = 00000000
15:27:36.862419 DEBUG:   XTRX PCI_3: SPI[0/1] 002fffff => 00000000
Detected LMS #0: 00000000
sergforce commented 5 years ago

Looks good! Have you tried to run any application or test_xtrx to see a data transmission?

sergforce commented 5 years ago

Can you test the latest version?

gustad commented 5 years ago

Thanks. It seems like the default gcc settings is a bit strict on this distro:

petter@rockpro64:~/projects/sdr/xtrx/xtrx_linux_pcie_drv.git$ gcc --version
gcc (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

petter@rockpro64:~/projects/sdr/xtrx/xtrx_linux_pcie_drv.git$ git rev-parse --short HEAD
a089414
petter@rockpro64:~/projects/sdr/xtrx/xtrx_linux_pcie_drv.git$ make
make -C /lib/modules/4.4.132-1075-rockchip-ayufan-ga83beded8524/build M=/home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git modules
make[1]: Entering directory '/usr/src/linux-headers-4.4.132-1075-rockchip-ayufan-ga83beded8524'
  CC [M]  /home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.o
/home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.c:675:20: warning: 'xtrx_msi_irq_single' defined but not used [-Wunused-function]
error, forbidden warning: xtrx.c:675
scripts/Makefile.build:283: recipe for target '/home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.o' failed
make[2]: *** [/home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git/xtrx.o] Error 1
Makefile:1471: recipe for target '_module_/home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git' failed
make[1]: *** [_module_/home/petter/projects/sdr/xtrx/xtrx_linux_pcie_drv.git] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.4.132-1075-rockchip-ayufan-ga83beded8524'
Makefile:10: recipe for target 'modules' failed
make: *** [modules] Error 2
sergforce commented 5 years ago

I updated everything, now it should work just fine