make '-j' parameter equivalent for fusesoc

[m-kru]

I am not sure if it applies to all tools, but for Vivado FuseSoc does not use any parallelism capabilities (Vivado -jobs parameter). I wanted to improve this, so I have added support in Edalize https://github.com/m-kru/edalize/commit/5ca3651e3745fca6e9531dc68753273f54c29f2b. It can be used like this:

There is one issue with this approach. The target contains information on not only what to do, but also on how to do it. Such thing as number of jobs has more to do with the machine you built the design on than with the design per se.

What would be more appropriate is the equivalent of make's -j parameter. One would add -j argument when fusesoc is called. Then, fusesoc would seamlessly forward the information about the parallelism to the EDA tool, hiding all the EDA tools details from the user.

The backend for that could be the same as it is right now, which means that defining number of jobs within the tool options in the .core file still would be possible, however the recommended way would be passing the -j argument.

In case of purely Makefile backends (such as GHDL) -j n would be simply forwarded to the make when it is called.

@olofk @imphil what do you think?

[olofk]

There is a similar thing with verilator where you can specify that the simulator model should be built for a fixed number of threads. In that case it's possible to specify this as e.g. verilator_options : [threads, 4] just like you did with the jobs parameter, and as you say it's not optimal because it depends on the machine you build it on

But! you're missing one thing :)

All tool options can be set already from the command line. So for the verilator example, you can run fusesoc run --target=<target> <core> --verilatior_options="-threads 4" and if you run fusesoc run --target=<yourtarget> <yourcore> --help on your design you will see that the jobs parameter is there. So you can actually do fusesoc run --target=<yourtarget> <yourcore> --jobs=4 with the patch you proposed.

This is of course not completely perfect, in case you create the project files one machine and run it on another, but perhaps it's good enough for your case?

[sjalloq]

This needs to be documented. I've been wondering how I pass different parameters to simulations for ages. Or have I just missed it somewhere?

[m-kru]

@olofk adding -j would make the user interface more unified. Right now user needs to remember that one tool requires jobs the other one threads and yet another one tasks. I think it all should be hidden under the -j.

What I value in FuseSoc is the fact, that it presents quite unified interface to different tools. If I had to remember what are the parameter names for each of them I would simply use make.

[olofk]

@sjalloq I'm really not sure if it has been properly documented. We should look into that. It's not good that users don't find it.

@m-kru I think it's good to unify options when we can, but it's not always that easy. The edalize backends tell fuseesoc which parameters they use and there are very few parameters that are applicable to all backends. Then there is the question whether to break out parameters separately or just send a whole options string to the tools. There is for example no separate jobs option for verilator, but I think you can do it by passing --make_options="-j". Then there's also the third complication that it's not super easy at this point to allow single-dash options like that since the options list is built up dynamically from the options that the backend exposes together with the parameters that the target implements

[m-kru]

The -j would be an option of FuseSoc, then it would create proper option for backends. In case of Makefile it would be j in case of Vivado jobs in case of Verilator threads etc. From users perspective the -j N would have following semantics "I do not care what you do, how you do it, and what backend you use - I just want it to be done with no more than N OS threads simultaneously".

[imphil]

I don't see a way of making "-j" work coherently for all supported simulators. Not because the perfect scenario isn't achievable, but because I don't even see a "standard" use case that we can all agree on.

Take the verilator example: what would "-j" do? Build the simulation with x parallel jobs (that's make -jx in Verilator build)? Build the simulation to support running in threaded mode with x parallel jobs? (that's --threads x).

On Vivado: Vivado picks sensible default values for parallel synthesis, DRC and other steps (see "Multithreading with the Vivado Tools" in UG904), which depend on a variety of factors. They are chosen because Xilinx knows rather well how their application scales on different platforms. A user might know better (and that's why those defaults can be overwritten), but it's hard to argue that FuseSoC could pick a better default value than Vivado.

I see this as a common pattern: tools should and do optimize to make compile and runtime as fast as possible (everyone wants that, including the marketing department). So they should and do pick sensible default values for parallel execution. If and when the user wants to override that, it's going to be a tool-specific override. FuseSoC already supports that (in some cases, edalize might need to pass on more options to tool runs, that's of course something that should be added as needed).

[wallento]

I agree with @imphil. Semantics can get messy and different backends perform different actions. In Linux I believe the best way to implement something useful from my point of view is:

• Add a -j style parameter that sets maximum parallelism
• Run fusesoc in a cgroup with cpulimits

Thats not really fully integrated, but in my opinion is the best next to -j N.

[m-kru]

@imphil

I don't see a way of making "-j" work coherently for all supported simulators. Not because the perfect scenario isn't achievable, but because I don't even see a "standard" use case that we can all agree on.

Take the verilator example: what would "-j" do? Build the simulation with x parallel jobs (that's make -jx in Verilator build)? Build the simulation to support running in threaded mode with x parallel jobs? (that's --threads x).

As I have written "I do not care what you do, how you do it, and what backend you use - I just want it to be done with no more than N OS threads simultaneously". As you have mentioned, some paralelizm is already handled by the tools automatically. However, when you call make with -j. Do you care what is done under the hood for particular targets?

On Vivado: Vivado picks sensible default values for parallel synthesis, DRC and other steps (see "Multithreading with the Vivado Tools" in UG904), which depend on a variety of factors. They are chosen because Xilinx knows rather well how their application scales on different platforms. A user might know better (and that's why those defaults can be overwritten), but it's hard to argue that FuseSoC could pick a better default value than Vivado.

This would not be included for the Vivado backend. It treats about maxThreads, which is handled automatically by the Vivado. In the first post I was mentioning jobs argument, that is not handled automatically. Vivado builds called from FuseSoc take much longer than they potentially can. I know, because I have tested. The main bottleneck is the OOC Synthesis which is done one by one currently.

I wonder how you use Fusesoc. At first I was delighted with it, but now I see that it is great for collecting source and dependency management, but I can do the build stage much better with Makefiles.

[wallento]

However, when you call make with -j. Do you care what is done under the hood for particular targets?

make just runs different make targets in parallel, it doesn't care for how many threads each of the target creates. So, it is exactly not what you want, right? If you have one target that calls vivado you can have arbitrary -j, it won't matter.

I wonder how you use Fusesoc. At first I was delighted with it, but now I see that it is great for collecting source and dependency management, but I can do the build stage much better with Makefiles.

I think what you describe is exactly what FuseSoC is build for. As I suggested above, I would see job management on a different layer. I currently work on a tiny FuseSoC remote job runner, maybe its more in this domain. Anyhow, not saying that it shouldn't be in FuseSoC, just a different level in my opinion..

[olofk]

As I see it, we're talking about two slightly different, but related, things. The first is exposing Vivado's jobs parameter. Normally I prefer to expose as few tool options as possible and instead set things in tcl files that are sourced by Vivado. But I'm all for having this as a special parameter so that it can be specified on the command-line since it's relevant to the machine that builds it, rather than the core being built. And if it can speed up builds, that's great. I don't want FuseSoC (or technically Edalize) to be a limiting factor here if we can avoid it. So, I'm happy to accept a patch that adds a jobs tool option to the Vivado Edalize backend

The other thing is to adding a -j FuseSoC parameter. That, we can't do. Over the years the responsibilities of FuseSoC vs Edalize has become clearer and this is definitely an Edalize parameter, which means that each Edalize backend must tell FuseSoC about this option. And here we have a first limitation that parameters must start with --. This is of course technically possible to fix but I won't do that because the gains are not high enough to justify the extra complexity. But at the same time, I definitely agree that the backends should be consistent in their parameter naming. We do this already in some other cases. Like the synth and pnr parameters. Every backend that can switch between several synth or p&r tools use the same name for this parameter. This is just a convention, but as the maintainer I'm requiring new backends to be consistent. Same thing will happen once the backends start implementing e.g. a GUI mode. I will not allow some backend to call it --gui and another to call it --user_interface. And for the jobs parameter, Vivado would be the first to support this parameter, but any other backend exposing a similar option would be required to call it --jobs as well. But that's as far as we will go when it comes to consistency.

And as @wallento mentioned, there is ongoing work to further separate FuseSoC and Edalize. This will make it even easier if you want to use FuseSoC for depdendency management but let your own Makefiles handle the build itself.

[olofk]

Also noticed that the question of exposing jobs has come up before https://github.com/olofk/fusesoc/issues/225 Let's get it fixed (in Edalize)

[m-kru]

@olofk I have prepared PR for edalize https://github.com/olofk/edalize/pull/222.