search

trabucayre / openFPGALoader

Universal utility for programming FPGA
https://trabucayre.github.io/openFPGALoader/
Apache License 2.0
1.16k stars 249 forks source link

Support multiple devices on JTAG chain #81

Open wohali opened 3 years ago

wohali commented 3 years ago

I have a need to program multiple MachXO2/3 parts on a single JTAG chain.

It doesn't look impossible to add this support...but I don't exactly have the time myself for this right now.

trabucayre commented 3 years ago

Hi. True: this limitation is a bit annoying. Adding code to allows to deal with a JTAG chain is already in my TODO list. My main worry is most of my boards have an onboard adapter. I must bypass this one to check if no regressions, with all FPGA.

stefanct commented 3 years ago

I have a few xilinx 7-based SoCs (Zynq, UltraScale) that would also require this to become potentially supported.

trabucayre commented 3 years ago

You have right. I usually program zynq's FPGA from Linux but it's a good way to add this feature. It's true for cycloneV Soc too.

trabucayre commented 3 years ago

Hi, I have created a new branch JTAG_chain with the start of multiple device support. Currentty only tested with cycloneV Soc (ARM + FPGA).

stefanct commented 3 years ago

I gave it a try and it does something but not the right thing apparently :)

I was able to add the required ID for the zynq on my zedboard and it is identified just fine if I am using the right --index-chain option which is 1 in this case (btw i guess it would make sense to add a default chain to fpga_model). However, after flashing the SRAM of the FPGA does not start up correctly AFAICT. This is without any activity on the ARM core. The same setup works fine with vivado. Any ideas?

./openFPGALoader -v -b zedboard --index-chain 1 xilinx_pulpissimo.bit
write to ram
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 2 devices
index 0:
  idcode   0x4ba00477
  type     ARM cortex A9
  irlength 4
index 1:
  idcode 0x23727093
  manufacturer xilinx
  model  7z020
  family zynq 7020
  irlength 6
File type : bit
Open file DONE
Parse file DONE
bitstream header infos
date: 2021/04/27
design_name: xilinx_pulpissimo
hour: 03:43:00
part_name: 7z020clg484
toolVersion: 2018.3
userID: 0XFFFFFFFF
load program
Flash SRAM: [===================================================] 100.000008%
Done
detach error -5 (LIBUSB_ERROR_NOT_FOUND)

(the detach error is benign AFAICT but is actually a bug: openfpgaloader shouldn't try to attach by itself if there was no drive attached to begin with. libftdi 1.5 got a AUTO_DETACH_REATACH_SIO_MODULE but I haven't checked if that can work with all the different variants to open the usb devices you use.) I also like that it really tries hard by doing 0.000008% more work than strictly necessary :D

trabucayre commented 3 years ago

for libftdi it's true. I have added by the past an hack to auto-reattach... But now, it's officially supported, so this piece of code doesn't make sense.

I have an zedboard too, so I able to try. But first, could you git pull your view, I have updated xilinx implementation to drop some functions who bypass the logic for multiple device.

index-chain is only required if you have more than one FPGA in the chain, for zynq the CPU is take into account to obtain some required informations but not as a potential target, maybe I need to improve doc for this topic.

trabucayre commented 3 years ago

I've updated the repository after sucessfully tested with my zedboard. Now you can use

openFPGALoader -b zeboard xxx.bit

Need to test all others devices (anlogic, gowin, lattice) to check, in first time if there are (or not) regression with single device and try to create a more complex setup to have more than one FPGA in the chain.

stefanct commented 3 years ago

I can confirm that this works. Thanks! My patches look a bit different regarding naming (e.g., clg484 is just the package for example so I left it out) and I have fixed a few minor things on the way. Feel free to pick what you like... The ZCU102 is untested yet. Will test that in the next days. https://github.com/trabucayre/openFPGALoader/compare/JTAG_chain...stefanct:JTAG_chain?expand=1

BTW if one uses bscane2 JTAG primitives in a design, should this be visible by probing with oFPGAl (if the config has already been active before the probe happens)? I don't think so but I am not sure why TBH. It does not show up in Vivado either.

stefanct commented 3 years ago

OK, that was too fast or I was too tired yesterday. I checked again after a full power cycle of the zedboard and my softcore does not come up after flashing with openfpgaloader. It seems to work consistently after vivado flashed it once successfully. Seems to me that some parts are not transferred by ofl that are by vivado and stay in place even when ofl flashes. Can you confirm?

At least it is faster (although I hope there is room for some further tweaking - but only if we can make the actual I/O faster by improving the USB handling. NB: the almost 0 CPU time of ofl):

openFPGAloader: real 0m6,311s user 0m0,276s sys 0m0,357s

vivado 2018.3: real 0m17,029s user 0m12,597s sys 0m1,218s

stefanct commented 3 years ago

JTAG frequency seems to help more than I thought: 30MHz (max) does not work reliably (not sure if bug, e.g., in the prescaler handling, or if the hardware cannot deal with it), but 15MHz works fine for me and gets it down to 2,5 seconds.

trabucayre commented 3 years ago

with digilent_hs2 I have seen this frequency limitation. It's not true for all probe based on ft2232 or ft232. Need to check if something is wrong. Could you point a not working design? Theorrically openFPGALoader do everything, but maybe some steps must be added to initialize something (RAM, PLL, ...) The basic Litex example works well so there is something in your design needs some extra configuration. Thanks

stefanct commented 3 years ago

I am presuming you are only referring to the respective bit file that I have attached. The design is based on https://github.com/pulp-platform/pulpissimo/ but instantiates two bscane2 instances to connect the soft-core's debug module to the ARM+FPGA JTAG chain. The other IP cores used are: two times clk_wiz and blk_mem_gen each. If you need more details please let me know.

bla.tar.gz

trabucayre commented 3 years ago

Usually to check behaviour I use simple design just to see if configuration is okay, bitstream's content sent correctly, etc.

For this specific issue it's no more a problem with multiple device but with some init. Have you somewhere the modified design? I have build .bit from pulpissimo, but now I need to find why SDK won't compile and how to use it.

Thanks

trabucayre commented 3 years ago

After some difficulties to build toolchain from pulpissimo, I have build the design and the hello example. openFPGALoader to load .bit in RAM openocd to load riscV CPU a FT232R to have a serial port. Result: it's works. Can you share you design? The problem seems about one (or more) extra blocks. Thanks

stefanct commented 3 years ago

You are too fast ;) To see my problem the SDK is actually not needed because it shows even without software running in the softcore: neither the DONE LED nor LD0 is lit up. (I have verified that the debug interface is dead too). Furthermore, I have now tested with more upstream-ish bitfiles that do use the same unmodified list of IPs and does not contain any other potentially problematic things - they just have some small modifications within the CPU pipeline. This still does not work!

That's a quite perplexing. I cannot share my designs easily but I'll look into that when I have more time and if we cannot figure it out. Can you please attach your bit file so that I can try that to rule out other possibilities? You did not mention which external JTAG adapter you used. It doesn't matter anyway but for completeness what is it?

trabucayre commented 3 years ago

You are too fast ;)

I don't know, I think I have spend a bit of time to be able to compile the toolchain :)

To see my problem the SDK is actually not needed because it shows even without software running in the softcore: neither the DONE LED nor LD0 is lit up. (I have verified that the debug interface is dead too).

I've redo flash, tried to change jumpers position and the result is always a success (usually I prefer being able to reproduce an issue...)

Furthermore, I have now tested with more upstream-ish bitfiles that do use the same unmodified list of IPs and does not contain any other potentially problematic things - they just have some small modifications within the CPU pipeline. This still does not work!

It's weird, I have used default design without modifications...

That's a quite perplexing. I cannot share my designs easily but I'll look into that when I have more time and if we cannot figure it out. Can you please attach your bit file so that I can try that to rule out other possibilities?

the design is here

You did not mention which external JTAG adapter you used. It doesn't matter anyway but for completeness what is it?

it's a bus blaster v1 with an FT2232 and buffers. I have used this board because it's avoid a potential risk of confusion between all interfaces.

stefanct commented 3 years ago

Not sure if that counts as good news but your bitstream does not work either. This is with your latest commit (fc08249) without any modifications.

trabucayre commented 3 years ago

I think no ... It's really weird.

Thanks

stefanct commented 3 years ago

AFAICT the output is basically the same as with my branch before - apart from the tool version because it's your bitfile and the missing "LIBUSB_ERROR_NOT_FOUND" string:

$ ./openFPGALoader -v -b zedboard gwenhael.bit 
write to ram
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 2 devices
index 0:
    idcode   0x4ba00477
    type     ARM cortex A9
    irlength 4
index 1:
    idcode 0x23727093
    manufacturer xilinx
    model  xc7z020
    family zynq
    irlength 6
File type : bit
Open file DONE
Parse file DONE
bitstream header infos
date: 2021/05/17
design_name: xilinx_pulpissimo
hour: 19:04:56
part_name: 7z020clg484
toolVersion: 2019.2.1
userID: 0XFFFFFFFF
load program
Flash SRAM: [===================================================] 100.000008%
Done
detach error -5

The MIO settings have been unchanged since I got the board into my hands (MIO[6 downto 2]): 01100, i.e., pll, sdcard (which I currently don't use), cascaded jtag So AFAICT one of the configurations you tested.

I am using Debian 10/Buster (mostly - as in there are some newer packages but this should not matter: all of the following library come from buster/main).

libusb-0.1-4     2:0.1.12-32
libusb-1.0-0     2:1.0.22-2
libusb-1.0-0-dev 2:1.0.22-2
libftdi1-2:amd64 1.4-1+b2
libftdi1-dev     1.4-1+b2

Here is some more info... The kernel is a 5.9 from buster-backports/main. My zedboard is revision D. There is a JTAG-related errata but I would presume that we can rule that one out since Vivado works. I have included some more information about my setup...

$ cmake --version
cmake version 3.16.3

$ git log -1 --oneline 
fc08249 (HEAD, origin/JTAG_chain) lattice: don't use read_write

$ cmake -v ..
-- The CXX compiler identification is GNU 8.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") 
-- Checking for module 'libftdi1'
--   Found libftdi1, version 1.4
-- Checking for module 'libusb-1.0'
--   Found libusb-1.0, version 1.0.22
-- Checking for module 'libudev'
--   Found libudev, version 241
-- Configuring done
-- Generating done
-- Build files have been written to: ...

I don't think there is much going on at compile time so here is just an excerpt of a single compiler call and the linker step:

/usr/bin/c++  -DATTACH_KERNEL -DDATA_DIR=\"/usr/local/share\" -DFTDI_VERSION=104 -DUSE_UDEV -DVERSION=\"v0.2.6\" -I/usr/include/libusb-1.0 -I/usr/include/libftdi1  -std=gnu++11 -o CMakeFiles/openFPGALoader.dir/src/ftdiJtagMPSSE.cpp.o -c .../openFPGALoader/src/ftdiJtagMPSSE.cpp

/usr/bin/c++    -rdynamic CMakeFiles/openFPGALoader.dir/src/anlogic.cpp.o
  CMakeFiles/openFPGALoader.dir/src/anlogicBitParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/anlogicCable.cpp.o
  CMakeFiles/openFPGALoader.dir/src/dfuFileParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/dirtyJtag.cpp.o
  CMakeFiles/openFPGALoader.dir/src/efinix.cpp.o
  CMakeFiles/openFPGALoader.dir/src/efinixHexParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/fx2_ll.cpp.o
  CMakeFiles/openFPGALoader.dir/src/ice40.cpp.o
  CMakeFiles/openFPGALoader.dir/src/ihexParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/spiFlash.cpp.o
  CMakeFiles/openFPGALoader.dir/src/rawParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/usbBlaster.cpp.o
  CMakeFiles/openFPGALoader.dir/src/epcq.cpp.o
  CMakeFiles/openFPGALoader.dir/src/svf_jtag.cpp.o
  CMakeFiles/openFPGALoader.dir/src/jedParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/display.cpp.o
  CMakeFiles/openFPGALoader.dir/src/jtag.cpp.o
  CMakeFiles/openFPGALoader.dir/src/ftdiJtagBitbang.cpp.o
  CMakeFiles/openFPGALoader.dir/src/ftdiJtagMPSSE.cpp.o
  CMakeFiles/openFPGALoader.dir/src/configBitstreamParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/ftdipp_mpsse.cpp.o
  CMakeFiles/openFPGALoader.dir/src/xilinx.cpp.o
  CMakeFiles/openFPGALoader.dir/src/main.cpp.o
  CMakeFiles/openFPGALoader.dir/src/latticeBitParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/gowin.cpp.o
  CMakeFiles/openFPGALoader.dir/src/device.cpp.o
  CMakeFiles/openFPGALoader.dir/src/lattice.cpp.o
  CMakeFiles/openFPGALoader.dir/src/progressBar.cpp.o
  CMakeFiles/openFPGALoader.dir/src/fsparser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/mcsParser.cpp.o
  CMakeFiles/openFPGALoader.dir/src/ftdispi.cpp.o
  CMakeFiles/openFPGALoader.dir/src/altera.cpp.o
  CMakeFiles/openFPGALoader.dir/src/bitparser.cpp.o
   -o openFPGALoader  -lusb-1.0 -lftdi1 -lusb-1.0 -ludev -lftdi1 -ludev 

$ lsusb -v -d 0403:6014

Bus 004 Device 060: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0403 Future Technology Devices International, Ltd
  idProduct          0x6014 FT232H Single HS USB-UART/FIFO IC
  bcdDevice            9.00
  iManufacturer           1 Digilent
  iProduct                2 Digilent USB Device
  iSerial                 3 210248686770
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              2 Digilent USB Device
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0001
  Self Powered

I have also attached the strace log. strace.zip

trabucayre commented 3 years ago

Thanks for all informations. lsusb and openFPGALoader dump matches. I have a rev D too. Tried to load the .bit through docker with buster, again successfully... The problem with the libusb is maybe because ftdi_sio has been blacklisted, or a udev rules said to not attach your FTDI. Just to test I've unloaded this driver, I've the same message with error -5, but again DONE and LD0 are ok.

I not see why this .bit is working here with two zedboards, and is not working for you

stefanct commented 3 years ago

Yes, I am not loading ftdi_sio because it interferes with what I usually want to do and I am using silicon labs (cp210x) as external UARTs if need be.

I captured the complete USB traffic using wireshark and tried staring at it (or more specifically at the output of tshark -T fields -e usb.endpoint_address -e usb.endpoint_address.direction -e usb.capdata -i - <ofl.pcapng >ofl.txt etc.) and they share a lot of data obviously but it is very much unaligned and has lots of non-matching packets and/or headers/footers as well. Without a wireshark dissector for FTDI/JTAG traffic this is not very productive.

Maybe a really small example that reduces the amount of USB traffic significantly would be make this attempt more sensible. Is that even possible? If not I think I am kinda out of ideas too.

trabucayre commented 3 years ago

Using wireshark is a good idea, I do this to check behaviour of some vendor's tools. The main problem is vendors, usually, does obfuscation (SRAM is programmed more than one time, using MPSSE's instructions to drive pins mixed with instructions to sent bit or byte). Bitstream size is always the same so I'm not sure it's a solution. I have some piece of python code to decode output from wireshark, but not for xilinx. I think I must adapt existing code for this target to compare.

I'm not astonish by different traffic between vivado and openFPGALoader, the real question is to see if something is missing in openFPGALoader. But I'm a bit perplexed to see this bitstream working here and not for your zedboard. Have you a second one (or any zynq based board) to test?

stefanct commented 3 years ago

I don't have immediate physical access to another zynq-based fpga. I am planning to test it on a zcu102 soon-ish - but only remotely for now. Further tests will have to wait. I could test a zybo with an artix-35t though but that probably won't tell us much unless it also does not work :) I'll just give up for now on the zedboard. It's not a big deal.

trabucayre commented 3 years ago

zcu102 is a big step :) zybo is a zynq too (7010) no?

I'm a bit frustrated to be unable to find why this didn't work. I will try to another zynq I have.

stefanct commented 3 years ago

Right, I meant Arty. We have some Zybos at the office too but I am almost exclusively in home office till I am fully vaccinated... The zcu102 shouldn't be that much of a step compared to the zedboard from the JTAG's perspective. I am currently only struggling with centos' stone-old libraries on the remote PC else I would have tested it already :)

trabucayre commented 3 years ago

centos: it's amazing to see how it difficult to avoid this old distribution... you can try arty, it's working. Maybe if this one didn't work too there is a big problem somewhere...

I work from my laboratory and for the vaccine... It's not for tomorrow... Consequently it's the opposite, all my hardware is at home, since I work in openFPGALoader in my free time, some boards are from my work but everytime tested at home.

stefanct commented 3 years ago

The arty works fine with ofl. ¯_(ツ)_/¯ I'll report back when I have news on the ultrascales.

trabucayre commented 3 years ago

It's a good news. It's prove openFPGALoader is able, on your computer, to program a xilinx device. Just a stupid question: have you something plugged in zedboard's FMC connector?

Thanks

stefanct commented 3 years ago

Nope, I know why you ask ;) I have only some PMODs in use for UART and an external soft-core reset (with a pullup). Apart from that only power and the JTAG USB is connected. I was even suspecting the USB cable but another one shows the same behavior :/

trabucayre commented 3 years ago

I have no more idea... :-/ It's just amazing