Build ghdl-yosys-plugin as a module

Building ghdl-yosys-plugin as part of Yosys is not recommended and might become deprecated: https://github.com/ghdl/ghdl-yosys-plugin#build-as-part-of-yosys-not-recommended. Instead, it is suggested to build it as an object, an load it in Yosys as a plugin.

However, in this repository ghdl-yosys-plugin is built as part of Yosys (https://github.com/open-tool-forge/fpga-toolchain/blob/main/scripts/compile_yosys.sh#L28). Moreover, Yosys is built without plugin support (https://github.com/open-tool-forge/fpga-toolchain/blob/main/scripts/compile_yosys.sh#L55).

Is this because of some limitation when building Yosys statically?

Hi, thanks for the warning that you might deprecate this method of building the plugin.

As per discussion in #4, yosys dynamic plugins aren't supported on Windows which is the main reason it's done this way.

The other complication would be that building it as a dynamic lib would then typically result in dependencies on other dynamic libs, which is what I've been trying to avoid so far... At some point I tried linking static libs into a dynamic lib and found that, at least on Debian/Ubuntu, they are typically not built with -fpic or -fpie so it's essentially incompatible. As far as I could tell, I'd probably have to build everything from source or change approach and start shipping dynamic binaries with relative rpath set.

So really, for now I'm not sure what a good alternative is. I'm happy to maintain the patch etc. downstream if you'd like to clean it up from your repo at least - unless there is also specific code needed to support a static plugin?

First off, the patch is going nowhere in the short term. As long as there are use cases which can only be solved through it, I'm pretty sure that Tristan will maintain it "active". My main concern is that building the plugin in Yosys is not tested in CI (https://github.com/ghdl/ghdl-yosys-plugin/actions). The usage as a module is tested only. Hence yosys.diff might become outdated, compared to ghdl.cc, which happened several times in the last months.

As per discussion in #4, yosys dynamic plugins aren't supported on Windows which is the main reason it's done this way.

This is enough for considering NOT deprecating yosys.diff in the short-mid term. In fact, this is what I expected to understand when I opened this issue.

The other complication would be that building it as a dynamic lib would then typically result in dependencies on other dynamic libs

My understanding about compilers is quite limited. From a user's perspective, I am aware of the following cases:

yosys -p "ghdl ....: ghdl-yosys-plugin built into Yosys. Yosys can be built either statically or dynamically. [Used in this repo]
yosys -m ghdl -p "ghdl ....: ghdl-yosys-plugin built independently as ghdl.so, and placed in Yosys' default location for plugins. [Used in ghdl/docker images]
yosys -m path/to/ghdl.so -p "ghdl ....: ghdl-yosys-plugin built independently as ghdl.so and placed in any location.

I wonder if it's possible to build ghdl-yosys-plugin statically (ghdl.a), and then use it as in 2 or 3. However, my understanding is that Yosys' on windows cannot use anything dynamically, no matter how you built the plugin. Hence, this would neither work.

At some point I tried linking static libs into a dynamic lib and found that, at least on Debian/Ubuntu, they are typically not built with -fpic or -fpie so it's essentially incompatible.

You might need to tell it explicitly. For building GHDL itself, there is a --default-pie configuration option, which overrides the default in the system.

As far as I could tell, I'd probably have to build everything from source or change approach and start shipping dynamic binaries with relative rpath set.

I would strongly recommend not to do so. The main benefit of this project is that everything is built statically, which theoretically allows distributing three main packages only (Windows, Linux and macOS). That is extremelly valuable for most new users to test examples (from ghdl, fomu, symbiflow, etc.) without dealing with installing or understanding how all the tools fit together. For most use cases it works very well already (see im-tomu/fomu-toolchain#20).

Conversely, if libs were dynamic, the Linux package would probably no longer be compatible with multiple distributions (see im-tomu/fomu-toolchain#16 indeed). For a dynamic approach, I believe that hdl/containers (moved from ghdl/docker) is a better solution. That is, provide a single env (Debian Buster or Ubuntu 20.04), build all the tools there and provide a package to users. Then users of that distributions can use it on their host. Others can use containers (which work on either Linux -docker or podman-, Windows -docker or WSL- or macOS -docker-). In fact, I envision this project and hdl/containers as complementary approaches that might share multiple build scripts in the near future. The static build would be the 'nightly' approach for most use cases, and containers would be the 'nightly' approach for users on the bleeding-edge of advanced features.

An alternative to containers would be Snap/Flatpak/AppImage, but I don't find any of those more appealing.

So really, for now I'm not sure what a good alternative is. I'm happy to maintain the patch etc. downstream if you'd like to clean it up from your repo at least - unless there is also specific code needed to support a static plugin?

Given the context, I think that the current approach is good enough. I don't think you need to maintain the patch downstream. However, it'd be very interesting if you could help with keeping it up to date. That is, let's add the patch to CI in ghdl-yosy-plugin, so we get faster feedback when breaking changes are added. At the same time, please report the bugs/issues you find there, instead of fixing them here. Hence, I'm not asking you to maintain the patch, but to ensure that this project is not significantly decoupled, if that makes sense.

On my side, I will think about how to best communicate these differences to users. The canonical CLI call is yosys -m ghdl -p ..., which is found in many Makefiles and scripts on the net. However, that fails with fpga-toolchain. We need unexperienced users to know that all they need to do is remove -m ghdl. At the same time, we need experienced users to write their Makefiles/scripts so that -m ghdl is included conditionally. @rodrigomelo9's Getting started with FOSS for FPGAs might be a good fit for these explanations/clarifications.

Ref ghdl/ghdl#1336

@edbordin, alpin3/ulx3s illustrates what I mean with fpga-toolchain and hdl/containers being complementary projects. @kost's solution is for ULX3S only, and the structure of the repo is not designed for generalisation/scalation. Yet, it provides essentially the same set of tools either as static packages or container images.

/cc @mithro

First off, the patch is going nowhere in the short term. As long as there are use cases which can only be solved through it, I'm pretty sure that Tristan will maintain it "active".

Sorry, I was thinking you were looking at this from the perspective of a GHDL maintainer! I see now you're looking at it in the context of fomu-toolchain.

I wonder if it's possible to build ghdl-yosys-plugin statically (ghdl.a), and then use it as in 2 or 3

To elaborate on what I was talking about, it's relatively easy to turn on -fpie or -fpic in the code we are already compiling. But we are also linking against a bunch of prebuilt libs so in practice this would probably mean building libgnat.a + all of its dependencies from source with -fpie or -fpic too. I believe they are deliberately turned off for static libs most of the time because it limits optimisations that can be done. pthreads had some other complications that I didn't fully understand when trying to mix dynamic and static.

An alternative to containers would be Snap/Flatpak/AppImage, but I don't find any of those more appealing.

I have also been wondering if something like this would make things easier since it would mean the build scripts become much simpler. It would solve my issues around iverilog/VPI (#34). The main problem I see with it is it only solves the problem on Linux. It looks like Snap supports multiple entrypoints via "aliases" which was another concern I had about AppImage. edit: it actually looks like snap is also available for os x...

The canonical CLI call is yosys -m ghdl -p ..., which is found in many Makefiles and scripts on the net. However, that fails with fpga-toolchain.

I have been meaning to at least make a note of this in the README here as a couple of users have already come asking about that. I really should get around to that... Although it would not be a clean solution, we could add a small patch to yosys to ignore -m ghdl...

Sorry, I was thinking you were looking at this from the perspective of a GHDL maintainer! I see now you're looking at it in the context of fomu-toolchain.

I ain't a maintainer of GHDL 😉 . Although I am the most active contributor, it accounts for 1-2% of ghdl/ghdl and 5-10% of the org. Therefore, Tristan is the only authoritative voice, specially regarding language support, simulation and synthesis. I know mostly about cosim, CI, running on ARM/Android, packaging, documentation, etc.

Nonetheless, I'm looking at this as a contributor of GHDL indeed. It is frustrating for me that VHDL seems to be ignored and/or dismissed in some open source hardware camps. I believe that's unfair and very misleading for newcomers, specially the ones in Europe. Fomu is the only board I know which is given for free in many conferences and hacker meetings. Hence, it is strategically important that users have a wider view of the ecosystem when they read Fomu's workshop. At the same time, I do have a Fomu, and I want to use it for testing VHDL and mixed HDL designs using GHDL: im-tomu/fomu-workshop#334 (using containers), im-tomu/fomu-workshop#338 (using fpga-toolchain).

To elaborate on what I was talking about, it's relatively easy to turn on -fpie or -fpic in the code we are already compiling. But we are also linking against a bunch of prebuilt libs so in practice this would probably mean building libgnat.a + all of its dependencies from source with -fpie or -fpic too.

In the context of GHDL, this was discussed in https://github.com/ghdl/ghdl/issues/640#issuecomment-418470167. That is a different use case that I didn't want to bring yet, but it is also related to fpie/fpic. It's about co-simulation of VHDL and Python. More precisely, executing Python functions from VHDL. See https://umarcor.github.io/ghdl-cosim/vhdl202x/ and https://umarcor.github.io/ghdl-cosim/vhdl202x/use-cases.html#executing-arbitrary-python-callbacks-from-vhdl.

I did not try co-simulation with the GHDL in fpga-toolchain yet. On Windows, MSYS2 packages allow it already, and containers can be used on any platform. See https://github.com/ghdl/ghdl-cosim/runs/1238489252?check_suite_focus=true. Hence, it is not critical for fpga-toolchain to support it, but I think it'd come very handy.

I believe they are deliberately turned off for static libs most of the time because it limits optimisations that can be done. pthreads had some other complications that I didn't fully understand when trying to mix dynamic and static.

I think that trading optimisations for portability (static builds) is sensible. However, those pthreads issues might be more difficult to deal with...

I have also been wondering if something like this would make things easier since it would mean the build scripts become much simpler. It would solve my issues around iverilog/VPI (#34). The main problem I see with it is it only solves the problem on Linux. It looks like Snap supports multiple entrypoints via "aliases" which was another concern I had about AppImage. edit: it actually looks like snap is also available for os x...

It seems that the issue with iverilog/VPI #34 might also trigger when trying to co-simulate with GHDL...

My conviction is that the solution for Windows is using MSYS2. MSYS2 is included in GitHub Actions and it will soon be the default bash shell on Windows environments. Build recipes (PKGBUILD) are plain bash scripts, and almost the same sources can be used for MSYS2 or Arch Linux. However:

Packaging all these tools for MSYS2 is an unaffordable effort for anyone to handle alone.
That would partially decouple Windows builds from the ones for GNU/Linux and/or macOS, which would increase the maintenance burden.

OTOH, build scripts for appimage/snap/flatpak are likely very similar to scripts for GNU/Linux containers.

I have been meaning to at least make a note of this in the README here as a couple of users have already come asking about that. I really should get around to that... Although it would not be a clean solution, we could add a small patch to yosys to ignore -m ghdl...

See ghdl/ghdl-yosys-plugin#74 and YosysHQ/yosys#1640.

Even if -m ghdl is ignored, some users might provide -m path/to/ghdl.so because they want to test the latest plugin with fpga-toolchain. That will probably fail too. Hence, instead of filtering -m ghdl, I believe it would be better to just change the error message. Instead of:

ERROR: This version of yosys is built without plugin support.

I'd propose:

ERROR: This version of yosys includes the following plugins already: ghdl. Using plugins dynamically is not supported in this build. See reference_to_the_readme.

BTW, just not to forget about it: the version reported by ghdl --version in fpga-toolchain is empty: GHDL 1.0-dev () [Dunoon edition]. That's probably because git is not available in the environment where you are building it. I need to check it carefully.

Recap:

ghdl --version was fixed in #65.
Yosys' error when -m ghdl is used is being changed in YosysHQ/yosys#2512.
Generating static builds is being discussed in ghdl/ghdl#1560 and ghdl/ghdl#1547.

Therefore, I'm closing this issue.

YosysHQ / fpga-toolchain

Build ghdl-yosys-plugin as a module #58