on-the-fly decompression and spi-over-jtag

[emard]

HI!

I suggest gzip on-the-fly decompression to upload bitstreams. Especially it's good for spi-over-jtag bitstreams database each one from 2MB becomes only 3.3KB with gzip -9

[trabucayre]

Hi It's not clear for me:

• You propose to store spiOverJtag bridge in gzip format, uncompress this file before loading in RAM ? or
• sending the gzip by spiOverJtag, uncompressing this one in the FPGA before writing the FLASH?

[emard]

I propose to support zlib on-the-fly decompression inside of openFPGALoader and for flashing first search for spioverjtag.bit.gz on-the-fly decompress .gz using zlib support and program it and then flash like normal.

It will significantly reduce spioverjtag bitstream size in the binary installation of openFPGALoader

and I also propose to support reading bitstream from standard input, so that user can do for example:

zcat bitstream.bit.gz | openFPGALoader

On 9/4/20, Gwenhael Goavec-Merou notifications@github.com wrote:

Hi It's not clear for me:

• You propose to store spiOverJtag bridge in gzip format, uncompress this file before loading in RAM ? or
• sending the gzip by spiOverJtag, uncompressing this one in the FPGA before writing the FLASH?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-687119184

[GbGp]

Xilinx FPGAs already supports a compressed bitstream format that can be loaded with openFPGAloader as usual. To enable this feature it should be sufficient to just set set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design] constraint. Other than saving space this can make the flashing operation slightly faster.

[emard]

HI

I know xilinx lattice and others support compressed bitstream, but additional compression efficiency (file even shorter) can be achieved by compressing that with gzip -9 and that was the reason I was asking for this feature.

Instead of this, alternative would be to support reading bitstream from unix pipe (stdin) so I can use for example

zcat bitstream.bit.gz | openFPGALoader

On 11/26/20, GpGb notifications@github.com wrote:

Xilinx FPGAs already supports a compressed bitstream format that can be loaded with openFPGAloader as usual. To enable this feature it should be sufficient to just set set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design] constraint. Other than saving space this can make the flashing operation slightly faster.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-734243437

[trabucayre]

I'm agree with both proposition. A small test with xilinx compression and/or gzip shows:	size	compression
2192129	without xilinx compression
236791	with xilinx compression
3390	without xilinx compression + gzip -9
15910	with xilinx compression + gzip -9

Clearly gzip applied on non compressed file is smaller, but with both the flashing time is faster + file small The question is to determine which library to use to be compatible with linux, macOS and windows.

For unix pipe, my problem is a bit different: I use file extension to determine file type (it's diffcult or impossible to determine by content itself), with this approach it's required to find a way to select correct bitstream parser without extension information

[emard]

Me as unix-based user am inclined to a function independent of filenames and file content, just have commandline options to control what is done.

There can be user-friendly way which just works but should be allow to completely be turned off.

I think imagemagick "convert" is one of counter-examples what should not happen, when a knowledge of filename extension, content and commandline options is all required for user to know in order to get wanted result

On 11/26/20, Gwenhael Goavec-Merou notifications@github.com wrote:

I'm agree with both proposition. A small test with xilinx compression and/or gzip shows:	size	compression
2192129	without xilinx compression
236791	with xilinx compression
3390	without xilinx compression + gzip -9
15910	with xilinx compression + gzip -9

Clearly gzip applied on non compressed file is smaller, but with both the flashing time is faster + file small The question is to determine which library to use to be compatible with linux, macOS and windows.

For unix pipe, my problem is a bit different: I use file extension to determine file type (it's diffcult or impossible to determine by content itself), with this approach it's required to find a way to select correct bitstream parser without extension information

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-734292687

[trabucayre]

It's true, sometime letting deduce behaviour automatically is not a good thing. I think adding option to provides file format it's a good idea and the only solution to allows using unix pipe.

I will let nonetheless automatic detection for backward compatibility.

If someone can tell me an gzip library compatible with linux, windows and macOS ?

[emard]

right, passing filename extension should make it for pipe

from google search "zlib" seems available on all 3 platforms but I have tried it only on linux

On 11/28/20, Gwenhael Goavec-Merou notifications@github.com wrote:

It's true, sometime letting deduce behaviour automatically is not a good thing. I think adding option to provides file format it's a good idea and the only solution to allows using unix pipe.

I will let nonetheless automatic detection for backward compatibility.

If someone can tell me an gzip library compatible with linux, windows and macOS ?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-735214221

[trabucayre]

My problem is to be sure for all 3 platforms :-) Currently the documentation to install under windows and macOS is missing, if at least I don't break everything that would be good :-)

[emard]

Hmmm I see the situation, however if in doubt of compatibility I suggest just to leave pipe function and rely on system to decompress to stdin. I'm not expert but after 30yr of development wynd0uz could have some solution to mimic pipes.

On 11/28/20, Gwenhael Goavec-Merou notifications@github.com wrote:

My problem is to be sure for all 3 platforms :-) Currently the documentation to install under windows and macOS is missing, if at least I don't break everything that would be good :-)

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-735216256

[trabucayre]

Sorry for delay. In a first time I have used solution suggested by @GbGp to reduce bitstream size (and reduce load time): the artix50t bitstream is compressed, others are left unchanged to avoid noise by updating all of them I'm currently doing some test to refactor some classes to be able to use stdin. For file type as argument, again it's in pipe: this way is sometime required (for xilinx when using .bin file it's not possible to deduce if it's simple raw data or a bitstream).

[emard]

If you can also compile bitstream for artix100, that's on my boards so this is directly testable here

On 12/9/20, Gwenhael Goavec-Merou notifications@github.com wrote:

Sorry for delay. In a first time I have used solution suggested by @GbGp to reduce bitstream size (and reduce load time): the artix50t bitstream is compressed, others are left unchanged to avoid noise by updating all of them I'm currently doing some test to refactor some classes to be able to use stdin. For file type as argument, again it's in pipe: this way is sometime required (for xilinx when using .bin file it's not possible to deduce if it's simple raw data or a bitstream).

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-741582038

[trabucayre]

xc7a100 bitstream is already available. No ?

[emard]

Yes I forgot, maybe it needs small gzip -9 only, from 3.8MB to 4KB

On 12/9/20, Gwenhael Goavec-Merou notifications@github.com wrote:

xc7a100 bitstream is already available. No ?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-741858146

[trabucayre]

Sorry for the long delay. openFPGALoader has now --file-type to force format (instead of using file extension) and pipe support. It's now possible to use cat or gzip | openFPGALoader [options] --file-type xxxx

Thanks

[emard]

HI

Hey this is a great news! I tested new pipe option at ulx3s and it works good and you have speeded up ft231x somehow speed is now few % faster than fujprog (was about 50% slower before)

next week I could try some other boards too

On 2/24/21, Gwenhael Goavec-Merou notifications@github.com wrote:

Sorry for the long delay. openFPGALoader has now --file-type to force format (instead of using file extension) and pipe support. It's now possible to use cat or gzip | openFPGALoader [options] --file-type xxxx

Thanks

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-784845999

[trabucayre]

Thanks for you feedback. Yes I've take time to improve bitbanging speedup (I'm not sure if it's possible to increase more).

I will reread this issue and the other one to see if I have lost some improvements. And need to find a universal library to uncompress file on the fly

[emard]

I also don't know if anything faster is possible. fujprog can't get much speed from ft231x/ft232r than the approx same speed what you get now and this is effectively 500kbps (not really 3Mbps as openfpgaloader reports). Maybe ftdi chip is initialized for 3Mbps serial speed but it bitbangs jtag much slower

jtag bitbang method is fragile with compatibility and can trigger odd ftdi behaviour for some pc motherboards usb interfaces and/or different platforms linux/mac/win so we can be happy if it just works reliably

On 2/25/21, Gwenhael Goavec-Merou notifications@github.com wrote:

Thanks for you feedback. Yes I've take time to improve bitbanging speedup (I'm not sure if it's possible to increase more).

I will reread this issue and the other one to see if I have lost some improvements. And need to find a universal library to uncompress file on the fly

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-785637277

[trabucayre]

The speed reported by openFPGALoader is internal speed (I have tested with a logic analyzer) to consume FIFO's content, this why it's used. But it's true, for the full bitstream transfer the communication and the small internal FIFO size, will reduce the real speed.

[inkdot7]

This is from #132 by @trabucayre:

My worry about spiOverJtag directory size is primarly to the download/clone step. Xilinx bitstream, with compress option enabled, have an acceptable size, but intel/altera ones must have compress option disabled, and consequently have a size more important. Most globally, due the potential number of devices this directory may become awfully heavy.

Concerning decompression at install step vs on-the-fly decompression:

• the first is the most simple. No openFPGALoader's internal modifications to do, but requires to have access to the correct tools (gunzip/xz) for all systems (possible for windows and mac ?)

• the second only requires to have correct library (again windows/mac ?), no modification at install step but maybe increase programming time due to decompress step (it's maybe a wrong question, maybe impact is really small compared to spi flash transactions).

With the need to have access to the tools, I think it would be more convenient to a user especially on windows/mac if it has been compiled into the code using a library, rather than openFPGALoader having to find the executable at runtime.

Another alternative might be a small (~20 line) custom C compression/decompression using the basic Lempel-Ziv strategy. This could then be included directly in the openFPGALoader sources. Here is one example:

https://github.com/andyherbert/lz1

(There is no license statement however, but I guess the author could be contacted; @andyherbert.) I also tried to do an implementation just from the description given. Such an approach seems to be able to reach a few kB for the .bit files, and 50kB - 200kB for .rbf. This is not worse than a factor of 2 or 3 compared to .gz for the .rbf files.

[emard]

To avoid binary heavyness, if they are on main source repository to gzip them first with 4K block size, low-ram decompression friendly I have a python code here https://github.com/emard/esp32ecp5/blob/master/gzip4k.py

Other option is to create another repository github.com/OpenFPGALoader-bin that contains spioverjtag binaries

On 11/15/21, Håkan Johansson @.***> wrote:

This is from #132 by @trabucayre:

My worry about spiOverJtag directory size is primarly to the download/clone step. Xilinx bitstream, with compress option enabled, have an acceptable size, but intel/altera ones must have compress option disabled, and consequently have a size more important. Most globally, due the potential number of devices this directory may become awfully heavy.

Concerning decompression at install step vs on-the-fly decompression:

• the first is the most simple. No openFPGALoader's internal modifications to do, but requires to have access to the correct tools (gunzip/xz) for all systems (possible for windows and mac ?)

• the second only requires to have correct library (again windows/mac ?), no modification at install step but maybe increase programming time due to decompress step (it's maybe a wrong question, maybe impact is really small compared to spi flash transactions).

With the need to have access to the tools, I think it would be more convenient to a user especially on windows/mac if it has been compiled into the code using a library, rather than openFPGALoader having to find the executable at runtime.

Another alternative might be a small (~20 line) custom C compression/decompression using the basic Lempel-Ziv strategy. This could then be included directly in the openFPGALoader sources. Here is one example:

https://github.com/andyherbert/lz1

(There is no license statement however, but I guess the author could be contacted; @andyherbert.) I also tried to do an implementation just from the description given. Such an approach seems to be able to reach a few kB for the .bit files, and 50kB - 200kB for .rbf. This is not worse than a factor of 2 or 3 compared to .gz for the .rbf files.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-968750968

[trabucayre]

It's true, The best solution, in all case is to compress bitstream in the repository to reduce size and download/clone time. Using a second repo dedicated to binary files may be a solution but without compression clone time will be the same as current solution, and it's maybe less user friendly since it's required to fetch submodule (but I keep this option in my mind).

When I've said "an external tool to uncompress file" it's not at runtime but at install time: all bitstream are compressed in repository and instead to simply copy files, bitstream are uncompressed to /usr/share/openFPGALoader.

To resume the two options (but maybe others solutions are possible. I'm, of course, open, to all advice):

• bitstream are extracted are install time using external tool, openFPGALoader is not modified to add uncompress support. This option have no impact to time to write bitstream
• bitstream are uncompressed at runtime using an external, or internal, library. This option may increase a bit global runtime (really? compared to the spi Flash protocol?) to write bitstream, but everything is known since the library become a required dependency.

First one has my favor (maybe I'm wrong) because it has no impact on runtime/speed.

Of course, it's possible to select one option and finally to move to another if, finally, it's not the best.

[emard]

Maybe the easiest option is to decompress spioverjtag at install time and let openFPGALoader use them uncompressed.

I find on-the-fly decompression inside openFPGALoader good to have linux is easy, but as we discussed earlier, cross-platform compatibility could bring problems if we have to rely on external library.

Maybe we can just include simple C++ file that only decompresses maybe something like this or similar (I haven't tried) https://bestofcpp.com/repo/richgel999-miniz-cpp-compression

On 11/15/21, Gwenhael Goavec-Merou @.***> wrote:

It's true, The best solution, in all case is to compress bitstream in the repository to reduce size and download/clone time. Using a second repo dedicated to binary files may be a solution but without compression clone time will be the same as current solution, and it's maybe less user friendly since it's required to fetch submodule (but I keep this option in my mind).

When I've said "an external tool to uncompress file" it's not at runtime but at install time: all bitstream are compressed in repository and instead to simply copy files, bitstream are uncompressed to /usr/share/openFPGALoader.

To resume the two options (but maybe others solutions are possible. I'm, of course, open, to all advice):

• bitstream are extracted are install time using external tool, openFPGALoader is not modified to add uncompress support. This option have no impact to time to write bitstream
• bitstream are uncompressed at runtime using an external, or internal, library. This option may increase a bit global runtime (really? compared to the spi Flash protocol?) to write bitstream, but everything is known since the library become a required dependency.

First one has my favor (maybe I'm wrong) because it has no impact on runtime/speed.

Of course, it's possible to select one option and finally to move to another if, finally, it's not the best.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-969310842

[inkdot7]

Runtime penalty for decompression would be very small. E.g. zcat of 233 kB .gz file containing all the .bin and .rbf files (to 31 MB) takes 0.14 s on a fairly old AMD Athlon. That would be ~230 MB/s.

Programming my EP4CE15 with 511 kB takes 1.2 s, i.e. 0.42 MB/s.

[trabucayre]

Programming time depends, partially, to the probe: arty @30MHz takes 0,735s. I think having one .gz by bitstream is more easy and well VCS compliant (this avoid to update all the time one file, but just add new one). Anyway, I think I must try both solutions (and maybe add them: one as primary solution, second as backup). But before, I must find which library is linux/windows/mac/BSD compatible. Or as mentionned previously using an internal library but this one needs to be maintained (maybe as an early solution).

[inkdot7]

I did not want to suggest to put all bitstreams in one compressed file. That was only to test the decompression speed.

[trabucayre]

Okay! Sorry to my misunderstood. And thanks for your tests!

[emard]

for spioverjtag library where files are compressed with known-to-work tool, zlib is old and stable so in ideal world there should be near 0-maintenance. Of course we want code with clean and compiler independent style.

Here's uzlib another maybe even simpler than miniz, probably there are more project in this direction

https://github.com/pfalcon/uzlib

On 11/17/21, Gwenhael Goavec-Merou @.***> wrote:

Okay! Sorry to my misunderstood. And thanks for your tests!

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-971258722

[trabucayre]

Will check/test both. Maybe @jeanthom, @kost and @umarcor are able to point some multiplatform libraries to test. Thanks

[kost]

I would say there are 3 ways: 1) standard zlib 2) using minimalistic single source file decompression and include it with openFPGAloder: one of my favourites is miniz: https://github.com/richgel999/miniz (make sure to use release version, not from github master/main branch) 3) using modern library like https://github.com/zlib-ng/zlib-ng

To be honest, i would prefer option 2) since it is quite straightforward and not pain for user to use and build from the source on any platform. Second best option is standard zlib (option 1). Using anything else on Linux/Mac/Unix systems other than libz could be problematic for end user since most of them already have that default lib. Windows have to install any lib to build anyway. I would choose 3) only if performance is critical and project is heavily dependant on compression/decompression.

Hope it helps!

[umarcor]

I would go with good old zlib. That is already used in several projects, such as GHDL, GTKWave, Yosys... all of which are known to work on GNU/Linux, Windows and macOS. Actually, the difference between vcd and vcdgz is the same discussion you are having here, but applied to waveforms instead of bitstreams.

[jeanthom]

I don't have any experience with gzip libraries, but regarding packaging concerns on macOS both zlib and zlib-ng are well supported.

[trabucayre]

Thanks to all! If zlib is the most available/multiplatform I will use it.

[trabucayre]

I have added the code to decompress gz files (works on linux, building for mingw is okay too). Currently:

• no bitstream are compressed (it's easy to compress manually for validation), but new bitstreams will in this format
• it's limited to intel/altera (these bitstreams can't be internally compressed), for xilinx I don't know if it's required since these are already compressed by the tool.
• a fallback is present: if .gz file is not found openFPGALoader will search for same file withouth .gz extension

macOS support has to be updated accordingly (currently hid and zlib aren't supported)

Thanks all!

[emard]

Great!

I find it great if on-the-fly decompression would be also possible with any normal bitstream, for example:

openFPGALoader -b ulx3s bitstream.bit.gz

On 11/25/21, Gwenhael Goavec-Merou @.***> wrote:

I have added the code to decompress gz files (works on linux, building for mingw is okay too). Currently:

• no bitstream are compressed (it's easy to compress manually for validation), but new bitstreams will in this format
• it's limited to intel/altera (these bitstreams can't be internally compressed), for xilinx I don't know if it's required since these are already compressed by the tool.
• a fallback is present: if .gz file is not found openFPGALoader will search for same file withouth .gz extension

macOS support has to be updated accordingly (currently hid and zlib aren't supported)

Thanks all!

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-978916610

[trabucayre]

In fact it's not really limited to spiOverJtag with altera:

openFPGALoader -b ulx3s bitstream.bit.gz --file-type bit

do the job. Have to modify the file type detection to check if extension is gzip or gz and to take into account second extension

[emard]

yes yes this works for "master" branch. right, when .gz is found at the end, small parser can check is there .bit .sof and similar

The .gz is really great because it's so efficient at compressing bitstreams, bitstream distributions will be very light using it

Thanx very much for great support

On 11/25/21, Gwenhael Goavec-Merou @.***> wrote:

In fact it's not really limited to spiOverJtag with altera:

openFPGALoader -b ulx3s bitstream.bit.gz --file-type bit

do the job. Have to modify the file type detection to check if extension is gzip or gz and to take into account second extension

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-979392678

[trabucayre]

it's fixed on master branch. Now gz extension is detected, real type too. I think I need to simplify a bit. This detection is done twice: in device.cpp and in configBitstreamParser.

[emard]

I can confirm on master branch that gzip -9 compressed bitstream works with on-the-fly decompression:

openFPGALoader -b ulx3s bitstream.bit.gz

On Fri, Nov 26, 2021 at 7:38 AM Gwenhael Goavec-Merou < @.***> wrote:

it's fixed on master branch. Now gz extension is detected, real type too. I think I need to simplify a bit. This detection is done twice: in device.cpp and in configBitstreamParser.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/trabucayre/openFPGALoader/issues/47#issuecomment-979730245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4K2RZSSRBGMAA6BGIZA33UN4TQFANCNFSM4QUO3HVA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[trabucayre]

Great to know that! Thanks!