Support for multi-file generators

[GCHQDeveloper560]

I'm experimenting with a generator that produces HDL packages containing constants defined in a core file. The current version uses Jinja templates. There are a total of three files at the moment: the Python script and VHDL and SystemVerilog templates. However, I'm currently having trouble figuring out the best way to include this in a repo like fusesoc-generators or use this from a FuseSoC core.

The CAPI2 generators element allows a command, but nothing like a fileset to include multiple files. With the Python script listed as the command the Jinja templates don't get copied to the build directory and aren't found.

The only alternative I've thought of at this point is that multi-file Python generators would need to be bundled into a Python package, but this seems like overkill for this case and doesn't cover other languages.

Am I missing a way to define a generator with multiple files? If not should this capbility be added, and if so how? Am I missing an easier alternative?

[Jbalkind]

Our approach to this has been to pass a list of arguments to the generator. You can see an example at the bottom of this file: https://github.com/getziadz/openpiton/blob/fusesoc-clean/piton/design/chip/tile/rtl/tile.core and the actual generator's parsing of the process_me list here: https://github.com/getziadz/openpiton/blob/fusesoc-clean/piton/tools/src/fusesoc/preproc.py

[benreynwar]

The generation script should write a yaml file in it's current working directory describing the files that it generated. See fusesoc/capi2/generator.py for a helper class that can do some of this for you if your generation script is in python.

On Mon, May 13, 2019 at 3:32 AM GCHQDeveloper560 notifications@github.com wrote:

I'm experimenting with a generator that produces HDL packages containing constants defined in a core file. The current version uses Jinja templates. There are a total of three files at the moment: the Python script and VHDL and SystemVerilog templates. However, I'm currently having trouble figuring out the best way to include this in a repo like fusesoc-generators or use this from a FuseSoC core.

The CAPI2 generators element allows a command, but nothing like a fileset to include multiple files. With the Python script listed as the command the Jinja templates don't get copied to the build directory and aren't found.

The only alternative I've thought of at this point is that multi-file Python generators would need to be bundled into a Python package, but this seems like overkill for this case and doesn't cover other languages.

Am I missing a way to define a generator with multiple files? If not should this capbility be added, and if so how? Am I missing an easier alternative?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/olofk/fusesoc/issues/273, or mute the thread https://github.com/notifications/unsubscribe-auth/AAC76BBPB47SNZWIQ4E7X6DPVE7VRANCNFSM4HMOPKBA .

[benreynwar]

Ah, I see your problem now. I have my generators as part of a python package so I never noticed this.

[GCHQDeveloper560]

Our approach to this has been to pass a list of arguments to the generator. You can see an example at the bottom of this file: https://github.com/getziadz/openpiton/blob/fusesoc-clean/piton/design/chip/tile/rtl/tile.core and the actual generator's parsing of the process_me list here: https://github.com/getziadz/openpiton/blob/fusesoc-clean/piton/tools/src/fusesoc/preproc.py

Thanks for the example! It looks like your templates are embedded in the source file as strings. I had thought about doing that, but the Jinja docs seem to discourage it and it doesn't seem to scale well.

Ah, I see your problem now. I have my generators as part of a python package so I never noticed this.

If I went this route would it make sense to add generators to the FuseSoC package, create a separate fusesoc-generators package (like the fusesoc/fusesoc-generators repo), or just make a package for this tiny module?

This obviously doesn't work for any future generators that aren't Python, so does adding support for a fileset or something similar for generators: make more sense?

[GCHQDeveloper560]

I've posted the specific example I was experimenting with at fusesoc/fusesoc-generators#1. The work-around here was to use the path where the script is being run ('argv[0]') to find the other files.

If this technique is reasonable and generalisable I'll close this issue.

[benreynwar]

Would using self.files_root help to get at the template files in the generator class?

[GCHQDeveloper560]

At least in this specific case self.files_root is in the examples directory, while the script and templates are one level higher. It seems like the location of the script might be a little more reliable, but I could also be doing something silly!

[benreynwar]

That makes sense. I think your approach is the right way to do this. The relationship between self.files_root, the script location, and the template location should be constant so using self.files_root, __file__, or argv[0] should all work fine.

[olofk]

There are probably Python purists who have strong feelings about the choice of self.files_root, __file__, or argv[0]. If things are working I think we better close this issue before they are flocking here and never lets us finish this