Is it possible to generate sequential blocks in runtime?

[zhuanhao-wu]

If I want to:

1. generate a state-machine dynamically based on the user input of states and transitions,
2. then translate the module into Verilog,

is there an easy way to accomplish this?

I have tried generating the AST / the string of the sequential block function, but the Verilog translator requires source code of the block, which is inaccessible.

I could also try writing a template python file and fill in the state transitions and wire declarations but this does not seem natural to me.

[cbatten]

Hmmm ... this is a great question. Not sure if there is an easy way to do this in PyMTL. I think you almost want is a DSL for specifying FSMs and then to be able to automatically generate the PyMTL which then can be translated into Verilog ... good feature request for the new version of PyMTL though :)

[zhuanhao-wu]

I think it would be good if the translation tool doesn't require the source code for introspection (just the syntax tree) or there is an option indicating we are willing to trade-off the readability of the Verilog against flexibility.

Thanks for your reply

[cbatten]

Ah I see what you are saying ... the translation tool needs the source code to display as a comment in the Verilog ... but since you are generating the AST there is no source code. Should be pretty easy to hack PyMTL not to try and access the src? Or here is an idea ... can you just add some dummy src? That way you don't have to change PyMTL?

[cbatten]

If you create a minimal py.test test case that fails (i.e., a minimal test case that creates an AST without source) then I would be happy to see if we can hack something where if the src is not present then it just doesn't try to display it? Just create a pull request with a minimum failing test case and we will take a look!

[zhuanhao-wu]

Hi, I've created a pull request.

And if I replace the AST with normal function declaration, the tests should pass.

Thank you.

[ptpan]

The source code is not required to correctly parse the AST -- we can skip the inspection of source code and use the AST of blocks instead. The problem is that the generated AST of the block (such as the one in #170) cannot capture the variables that appear outside the definition of the block but inside the _ init _ function (i.e. in the closure of the block). This is due to the fact that the block is generated and compiled as a single function and compile() has no way to tell whether the free variables in the block are global or not. PyMTL needs this information to understand the use of different variables in the blocks (for example, the 's' variable is a free variable that appears in the block's closure, but a dynamically generated block cannot capture this!).

[cbatten]

THanks @ptpan I am not sure if there is an easy fix. Question for @ptpan and @zhuanhao-wu, does the generated AST actually simulate in pure-PyMTL? So is the issue only with translation or does simulating in PyMTL not work either? @zhuanhao-wu any thoughts on how we might be able to address this?

[ptpan]

No, the generated AST does not simulate in PyMTL. The simulation tool needs to detect whether the .next/.value is missing, where the closure information is used.

[cbatten]

Oh ... I thought it simulated. This overall approach to generating PyMTL code seems far less useful if you can't simulate it in PyMTL ... @ptpan any thoughts on a different way to achieve the same goal that @zhuanhao-wu originally wanted in a way that can simulate and translate? Basically the idea is to programmatically generate update blocks ... maybe if we can't support this in PyMTLv2 we can at least think about it for PyMTLv3?

[zhuanhao-wu]

One way I could think of is having __new__() of the Model class create a temporary python file , import the temporary file and create an instance out of it, something like:

class ModelGenerator(pymtl.Model):
  def __new__(cls, stmt, ...):
    # parse stmt into python code, maybe with the help of AST
    # load the source file of ModelGenerator, replace the PLACEHOLDER inside __init__() with the stmt python code, and remove __new__() method
    # copy the modified source code into another file, say inst.py, then
    exec('import inst')
    exec('real_inst = inst.ModelGenerator(stmt, ...)')
    return real_inst

  def __init__(s, ...):
    # create necessary ports
    # PLACEHOLDER for state machine logic

When used, it will be

from xxx import ModelGenerator
model = ModelGenerator(stmt, ...)

This should work on the premise that source code must be present.

[cbatten]

ah I see -- so to generate the complete source code. might be something to try?

[ptpan]

Just had a talk with @jsn1993 and we can make this work with little modification to PyMTL v2. We need to make the following changes to PyMTL v2:

1. If the inspect module fails to retrieve the source code of a model, simply return null string and line number (or maybe some meaningful text).
2. Use the AST provided by the user instead of parsing from the source code.

The user needs to do the following to make it work:

1. Add a new field (maybe called 'ast') to the dynamically generated block function which stores its (dynamically generated) AST.
2. Provide the union of global and local name space to exec() so that the free variables inside the dynamic block can be found. An example:

   class A(Model):
   def __init__(s):
   s.in_ = InPort(2)
   s.out = OutPort(2)
   tree = ... # dynamically generate the AST here, suppose the block is 'logic()'!
   exec(compile(tree, filename='<ast>', mode='exec')) in globals().update(locals())
   logic.ast = tree # 'logic' is the name of the block, .ast is the field that Pymtl will extract AST from

   With the above code, we can successfully simulation & translate the A model with minor modification to PyMTL.

[ptpan]

This works because PyMTL treats the global variables and the variables in the closure of the block similarly: PyMTL extracts the object from either the global name space or the closure. It is true that some variables might appear in the closure rather than the global space, but taking them as global does not break the simulation or translation pass...

[cbatten]

neat! Maybe @zhuanhao-wu can give it a try? I think we need to patch PyMTLv2 first to do 1 and 2? Maybe we can do that real quick?

[cbatten]

By "we" I mean @ptpan :)