support for NetFPGA Sume

[tudang]

I've built connectal from source and installed drivers, fpgajtag. After building Echo example for NetFPGA Sume target, I ran make run.nfsume and received the following error:

fpgajtag: elf input file, len 2170721 class 2
fpgajtag: unzip input file, len 1751914
fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700
[init_fpgajtag] id 3691093 from file does not match actual id 3410003
subprocess pid 6326 completed status=ff00 255
make[1]: *** [run] Error 255
make: *** [run.nfsume] Error 2

What should I do to resolve it?

[hanw]

Use vivado to program the bit file instead. I have seen this error on some sume boards, it is related to fpgajtag and the fact that there is a CPLD and FPGA on the same jtag chain.

[hanw]

You can also install Digilent Adept and use the command line tool : djtgcfg

[jameyhicks]

As Han said, this is a problem with fpgajtag, which is not handling the CPLD properly on NFSUME.

After you program the FPGA with vivado or djtgcfg,

1. run pciescanportal to configure the FPGA's PCIE registers
2. set the environment variable NOPROGRAM=1 when running the application so that connectal will not try to program the FPGA.

We'll either fix fpgajtag or add options for connectal to run vivado or djtgcfg to program the fpga.

[tudang]

Thank you for your instructions.

I program the FPGA using vivado in batch mode. Then I do the following steps:

• Run the application

 NOPROGRAM=1 ./ubuntu.exe 
subprocess pid 754 completed status=0 0
[initPortalHardwareOnce:256] fd 5 len 0
[checkSignature:154] read status from '/dev/connectal' was only 0 bytes long
FwdTestIndication_handleMessage: unknown channel 0xfffffffe
...
FwdTestIndication_handleMessage: too many bogus indications, exiting

• Run pciescanportal and then run the application

$pciescanportal
Running /usr/bin/pciescan.sh
+ for devid in 1be7:c100 1be7:b100
++ lspci -d 1be7:c100
++ sed -e 's/ .*//'
+ BLUEDEVICE=02:00.0
+ '[' 02:00.0 '!=' '' ']'
+ sh -c 'echo 1 >/sys/bus/pci/devices/0000:02:00.0/remove'
+ for devid in 1be7:c100 1be7:b100
++ lspci -d 1be7:b100
++ sed -e 's/ .*//'
+ BLUEDEVICE=
+ '[' '' '!=' '' ']'
+ sleep 1
+ lsmod
+ grep -q pcieportal
+ rmmod pcieportal
+ lsmod
+ grep -q bluenoc
+ sleep 1
+ sh -c 'echo 1 >/sys/bus/pci/rescan'
+ sleep 1

$ NOPROGRAM=1 ./ubuntu.exe
ubprocess pid 784 completed status=0 0
[initPortalHardwareOnce:256] fd 5 len 0
[checkSignature:154] read status from '/dev/connectal' was only 0 bytes long
Failed to open /dev/portal_1_7 fd=-1 errno=2
init_portal_internal: failed to initialize Portal portal_1_7

• Restart and run the application

subprocess pid 3071 completed status=0 0
[initPortalHardwareOnce:256] fd 5 len 0
[checkSignature:154] read status from '/dev/connectal' was only 0 bytes long
Requested main clock frequency 250.00, actual clock frequency  0.00 MHz status=-1 errno=0
Saying 42
[call_say:47] 42
heard an echo: 42
[call_say:47] 210
heard an echo: 210
[call_say:47] 714
heard an echo: 714
[call_say:47] 3906
heard an echo: 3906
TEST TYPE: SEM

My question: Is there anyway to avoid restarting the server to run the application properly like last step?

[jameyhicks]

Because the BIOS does part of the PCIe configuration process, you will have to restart the server once after programming the FPGA with a Connectal application.

After that, just running pciescanportal after reprogramming will be enough. Once the PCIe base address registers have been programmed by the BIOS, the kernel is able to reprogram them if directed to rescan the driver, which is what pciescanportal does.

If you program the FPGA with a design that uses different BAR sizes, then you'll have to reboot again.

We've looked at the pcie drivers and the chipset specs, but were unable to get around rebooting once in a while, but most of the time it is not required.

On Fri, Mar 4, 2016 at 10:40 AM Tu Dang notifications@github.com wrote:

Thank you for your instructions.

I program the FPGA using vivado in batch mode. Then I do the following steps:

• Run the application

  NOPROGRAM=1 ./ubuntu.exe subprocess pid 754 completed status=0 0 [initPortalHardwareOnce:256] fd 5 len 0 [checkSignature:154] read status from '/dev/connectal' was only 0 bytes long FwdTestIndication_handleMessage: unknown channel 0xfffffffe ... FwdTestIndication_handleMessage: too many bogus indications, exiting

• Run pciescanportal and then run the application

$pciescanportal Running /usr/bin/pciescan.sh

• for devid in 1be7:c100 1be7:b100 ++ lspci -d 1be7:c100 ++ sed -e 's/ .*//'
• BLUEDEVICE=02:00.0
• '[' 02:00.0 '!=' '' ']'
• sh -c 'echo 1 >/sys/bus/pci/devices/0000:02:00.0/remove'
• for devid in 1be7:c100 1be7:b100 ++ lspci -d 1be7:b100 ++ sed -e 's/ .*//'
• BLUEDEVICE=
• '[' '' '!=' '' ']'
• sleep 1
• lsmod
• grep -q pcieportal
• rmmod pcieportal
• lsmod
• grep -q bluenoc
• sleep 1
• sh -c 'echo 1 >/sys/bus/pci/rescan'
• sleep 1

$ NOPROGRAM=1 ./ubuntu.exe ubprocess pid 784 completed status=0 0 [initPortalHardwareOnce:256] fd 5 len 0 [checkSignature:154] read status from '/dev/connectal' was only 0 bytes long Failed to open /dev/portal_1_7 fd=-1 errno=2 init_portal_internal: failed to initialize Portal portal_1_7

• Restart and run the application

subprocess pid 3071 completed status=0 0 [initPortalHardwareOnce:256] fd 5 len 0 [checkSignature:154] read status from '/dev/connectal' was only 0 bytes long Requested main clock frequency 250.00, actual clock frequency 0.00 MHz status=-1 errno=0 Saying 42 [call_say:47] 42 heard an echo: 42 [call_say:47] 210 heard an echo: 210 [call_say:47] 714 heard an echo: 714 [call_say:47] 3906 heard an echo: 3906 TEST TYPE: SEM

My question: Is there anyway to avoid restarting the server to run the application properly like last step?

— Reply to this email directly or view it on GitHub https://github.com/cambridgehackers/connectal/issues/112#issuecomment-192327152 .

[jameyhicks]

I meant to include a link to the documentation for NOPROGRAM:

http://www.connectal.org/doc/current/ref/makefile.connectal.build.html?highlight=noprogram#envvar-NOPROGRAM

On Fri, Mar 4, 2016 at 11:00 AM Jamey Hicks jamey.hicks@gmail.com wrote:

Because the BIOS does part of the PCIe configuration process, you will have to restart the server once after programming the FPGA with a Connectal application.

After that, just running pciescanportal after reprogramming will be enough. Once the PCIe base address registers have been programmed by the BIOS, the kernel is able to reprogram them if directed to rescan the driver, which is what pciescanportal does.

If you program the FPGA with a design that uses different BAR sizes, then you'll have to reboot again.

We've looked at the pcie drivers and the chipset specs, but were unable to get around rebooting once in a while, but most of the time it is not required.

On Fri, Mar 4, 2016 at 10:40 AM Tu Dang notifications@github.com wrote:

Thank you for your instructions.

I program the FPGA using vivado in batch mode. Then I do the following steps:

• Run the application

  NOPROGRAM=1 ./ubuntu.exe subprocess pid 754 completed status=0 0 [initPortalHardwareOnce:256] fd 5 len 0 [checkSignature:154] read status from '/dev/connectal' was only 0 bytes long FwdTestIndication_handleMessage: unknown channel 0xfffffffe ... FwdTestIndication_handleMessage: too many bogus indications, exiting

• Run pciescanportal and then run the application

$pciescanportal Running /usr/bin/pciescan.sh

• for devid in 1be7:c100 1be7:b100 ++ lspci -d 1be7:c100 ++ sed -e 's/ .*//'
• BLUEDEVICE=02:00.0
• '[' 02:00.0 '!=' '' ']'
• sh -c 'echo 1 >/sys/bus/pci/devices/0000:02:00.0/remove'
• for devid in 1be7:c100 1be7:b100 ++ lspci -d 1be7:b100 ++ sed -e 's/ .*//'
• BLUEDEVICE=
• '[' '' '!=' '' ']'
• sleep 1
• lsmod
• grep -q pcieportal
• rmmod pcieportal
• lsmod
• grep -q bluenoc
• sleep 1
• sh -c 'echo 1 >/sys/bus/pci/rescan'
• sleep 1

$ NOPROGRAM=1 ./ubuntu.exe ubprocess pid 784 completed status=0 0 [initPortalHardwareOnce:256] fd 5 len 0 [checkSignature:154] read status from '/dev/connectal' was only 0 bytes long Failed to open /dev/portal_1_7 fd=-1 errno=2 init_portal_internal: failed to initialize Portal portal_1_7

• Restart and run the application

subprocess pid 3071 completed status=0 0 [initPortalHardwareOnce:256] fd 5 len 0 [checkSignature:154] read status from '/dev/connectal' was only 0 bytes long Requested main clock frequency 250.00, actual clock frequency 0.00 MHz status=-1 errno=0 Saying 42 [call_say:47] 42 heard an echo: 42 [call_say:47] 210 heard an echo: 210 [call_say:47] 714 heard an echo: 714 [call_say:47] 3906 heard an echo: 3906 TEST TYPE: SEM

My question: Is there anyway to avoid restarting the server to run the application properly like last step?

— Reply to this email directly or view it on GitHub https://github.com/cambridgehackers/connectal/issues/112#issuecomment-192327152 .

[tudang]

I understand now. I don't know fixing fpgajtag or add options in connectal to program netfpga is easier. I have a tcl script to program the board using vivado. How could I contribute?

[jameyhicks]

fpgajtag has been updated to use a lower JTAG clock rate, which seems to solve the programming problem on nfsume. I tagged this fpgajtag version v16.03.1 and I'm building ubuntu packages for it now.

Please give it a try. If it works we can close this issue.

[tudang]

I've updated the fpgajtag repo and run make inside it. After rebuilding Echo example, make run.nfsume produces:

echo$ make run.nfsume
make -C nfsume --no-print-directory run
obj/Makefile:1068: warning: overriding commands for target `verilog/mkPcieEndpointX7.v'
obj/Makefile:980: warning: ignoring old commands for target `verilog/mkPcieEndpointX7.v'
obj/Makefile:1140: warning: overriding commands for target `verilog/mkPcieEndpointX7.v'
obj/Makefile:1068: warning: ignoring old commands for target `verilog/mkPcieEndpointX7.v'
obj/Makefile:1471: warning: overriding commands for target `verilog/mkDdr3.v'
obj/Makefile:867: warning: ignoring old commands for target `verilog/mkDdr3.v'
/opt/connectal/scripts/run.pcietest bin/ubuntu.exe 
run.pcie parameters are: bin/ubuntu.exe
buffer /home/danghu/workspace/echo/nfsume/bin/ubuntu.exe
fpgajtag: elf input file, len 1123028 class 2
fpgajtag: unzip input file, len 749505
fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700
[init_fpgajtag] id 3691093 from file does not match actual id 3410003
subprocess pid 10247 completed status=ff00 255
make[1]: *** [run] Error 255
make: *** [run.nfsume] Error 2

$ fpgajtag -l
fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700; IDCODE:  3410003  487c003

Do I miss some steps?

[jameyhicks]

which fpgajtag? Is it the one from the ubuntu package? if so, apt-get update; apt-get install fpgajtag

On Mon, Mar 7, 2016 at 3:42 PM Tu Dang notifications@github.com wrote:

I've updated the fpgajtag repo and run make inside it. After rebuilding Echo example, make run.nfsume produces:

echo$ make run.nfsume make -C nfsume --no-print-directory run obj/Makefile:1068: warning: overriding commands for target verilog/mkPcieEndpointX7.v' obj/Makefile:980: warning: ignoring old commands for targetverilog/mkPcieEndpointX7.v' obj/Makefile:1140: warning: overriding commands for target verilog/mkPcieEndpointX7.v' obj/Makefile:1068: warning: ignoring old commands for targetverilog/mkPcieEndpointX7.v' obj/Makefile:1471: warning: overriding commands for target verilog/mkDdr3.v' obj/Makefile:867: warning: ignoring old commands for targetverilog/mkDdr3.v' /opt/connectal/scripts/run.pcietest bin/ubuntu.exe run.pcie parameters are: bin/ubuntu.exe buffer /home/danghu/workspace/echo/nfsume/bin/ubuntu.exe fpgajtag: elf input file, len 1123028 class 2 fpgajtag: unzip input file, len 749505 fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700 [init_fpgajtag] id 3691093 from file does not match actual id 3410003 subprocess pid 10247 completed status=ff00 255 make[1]: * [run] Error 255 make: * [run.nfsume] Error 2

$ fpgajtag -l fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700; IDCODE: 3410003 487c003

Do I miss some steps?

— Reply to this email directly or view it on GitHub https://github.com/cambridgehackers/connectal/issues/112#issuecomment-193441195 .

[jankcorn]

I think that you are still running the older fpgajtag.

(the fpgajtag -l value read out is incorrect 3410003 should be 3691093)

jca

On Mon, Mar 7, 2016 at 3:42 PM, Tu Dang notifications@github.com wrote:

I've updated the fpgajtag repo and run make inside it. After rebuilding Echo example, make run.nfsume produces:

echo$ make run.nfsume make -C nfsume --no-print-directory run obj/Makefile:1068: warning: overriding commands for target verilog/mkPcieEndpointX7.v' obj/Makefile:980: warning: ignoring old commands for targetverilog/mkPcieEndpointX7.v' obj/Makefile:1140: warning: overriding commands for target verilog/mkPcieEndpointX7.v' obj/Makefile:1068: warning: ignoring old commands for targetverilog/mkPcieEndpointX7.v' obj/Makefile:1471: warning: overriding commands for target verilog/mkDdr3.v' obj/Makefile:867: warning: ignoring old commands for targetverilog/mkDdr3.v' /opt/connectal/scripts/run.pcietest bin/ubuntu.exe run.pcie parameters are: bin/ubuntu.exe buffer /home/danghu/workspace/echo/nfsume/bin/ubuntu.exe fpgajtag: elf input file, len 1123028 class 2 fpgajtag: unzip input file, len 749505 fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700 [init_fpgajtag] id 3691093 from file does not match actual id 3410003 subprocess pid 10247 completed status=ff00 255 make[1]: * [run] Error 255 make: * [run.nfsume] Error 2

$ fpgajtag -l fpgajtag: Digilent:Digilent USB Device:210301763224; bcd:700; IDCODE: 3410003 487c003

Do I miss some steps?

— Reply to this email directly or view it on GitHub https://github.com/cambridgehackers/connectal/issues/112#issuecomment-193441195 .

[tudang]

I forget to run make install. After updating, there is another error while make run.nfsume:

Error: initPortalHardwareOnce: failed to open /dev/connectal, exiting
make[1]: *** [run] Error 255
make: *** [run.nfsume] Error 2

I change permission on /dev/connectal and re-execute make run, then the problem is solved.

[jameyhicks]

Excellent! Sounds like the fpgajtag problem is solved.

There is a udev rule file that is supposed to change the protection mode on /dev/connectal. This should be installed when you install connectal (either through make install or by installing the Ubuntu package).

On Tue, Mar 8, 2016 at 4:04 AM Tu Dang notifications@github.com wrote:

I forget to run make install. After updating, there is another error while make run.nfsume:

Error: initPortalHardwareOnce: failed to open /dev/connectal, exiting make[1]: * [run] Error 255 make: * [run.nfsume] Error 2

I change permission on /dev/connectal and re-execute make run, then the problem is solved.

— Reply to this email directly or view it on GitHub https://github.com/cambridgehackers/connectal/issues/112#issuecomment-193675096 .