Program Generator Waypoint

[ginty]

This PR is a periodic merge of the latest development on the program generator. At this point, the prb1 test flow from O1 can be fully executed on the front end, which means that the frontend (app) API is pretty capable and close to its final form. This is the main body of the prb1 flow written for O2: https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/test_apps/python_app/example/flows/o1_testcases/components/prb1_main.py

This also includes a number of general infrastructure improvements that are applicable to other parts of Origen, these are documented below.

Notes on ProgGen Arch

This contains a change in approach to how the program generator is implemented. Until now the generator actually constructed test objects and maintained a lot of state while executing the flow source files. This was starting to make life really complicated on the front end as it basically tried to construct much of the test program in real time (like how O1 did it).

With this update the prog gen is now completely stateless on the frontend and all the frontend APIs are concerned with is building the flow AST. Everything goes in the AST, even setting attributes on a test object, e.g. my_test_method.some_attr = "some_val" are simply converted into AST entries. The tester-specific backends will then walk the ASTs and build the final program, using whatever intermediate representations they need to get their job done.

The targets are rendered concurrently and the concurrency piece is already implemented, Rust really did enable easy concurrency, see here for where the threads are created - https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/pyapi/src/prog_gen/mod.rs#L61

The frontend prog API is implemented in pyapi/src/prog_gen/interface.rs which deals with all the commonly available APIs for any tester.

The tester-specific APIs are here:

• pyapi/src/tester_apis/igxl/prog_gen_api.rs
• pyapi/src/tester_apis/v93k/prog_gen_api.rs

Each one of those has access to the current derivative tester if it needs to handle them differently.

It is envisaged that pat_gen_api.rs files following the same approach will be added in due course.

The frontend functions call a back end flow API from where all test program nodes are generated and inserted into the current flow AST - https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/origen/src/prog_gen/flow_api.rs

It is envisaged that if (for example) we were to support flow conversion in future, then a flow reader would bypass the frontend API and speak to this backend API directly. To support such things I've ensured here that the frontend does not create any nodes itself and is only a proxy for the backend.

On the backend, origen::FLOW returns a flow manager object which is directly analogous to (in fact, implements pretty much the same API) the origen::TEST for patgen.

The backend generation piece is still in a state of flux so I won't cover it now since it is subject to change. Everything that has been documented here in this PR is considered final-intent.

If you want to try it you can run:

origen g example/flows/o1_testcases/prb1.py -t targets/dut/o1_dut0.py -t targets/tester/v93k_smt7.py

Useful Additions / Changes To Be Aware Of

tester().ast

This method can be called from the command line to see the current AST, it works for both prog gen and patgen and will automatically pick the correct AST.

If you want to see the AST, stick a breakpoint somewhere in the Python code (import pdb; pdb.set_trace()) and call this method.

Tester Specific Blocks

I changed tester().specific("j750") to be tester().eq("j750"), mainly because I came across the need to effectively do an else branch when converting some O1 prog gen test code. So you can now do that like:

with tester().eq("j750"):
  # Will only be rendered for J750 tester

with tester().neq("j750"):
  # Will be rendered for everything else

I also added the named tester types "all", "igxl" and "v93k" which will effectively resolve to true for the derivative tester types that you would expect.

I also removed the enforcement that with tester() blocks can only select a subset of an existing selection. This proved to make application code much harder to write - e.g. say in the flow or a pattern you have a test that you only want to call for a specific tester, but then within that block you call a function to generate it and that function has its own tester-specific clauses.

A processor has been added to resolve these blocks for a specific tester in the backend - https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/origen/src/generator/processors/target_tester.rs

This can be used for patgen too and should in most cases be the first processor to run, here's an example of invoking it: https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/origen/src/prog_gen/advantest/smt7/mod.rs#L23

Block/DUT Attributes

The attributes.py files in blocks are now hooked up and working, here is how to define an attribute: https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/test_apps/python_app/example/blocks/dut/attributes.py#L1

These will inherit and override from parent/child block definitions as you would expect.

To access an attribute:

dut.has_margin0_bug   # => False
dut.attributes["has_margin0_bug"]   # => False

Python Source Filename/Line Number Tracking

The way that the prog gen now works means that application issues may be discovered during final render and these will need to be tracked back to a file name and line number in the application.

I added a new caller utility module to help with this here: https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/pyapi/src/utility/caller.rs

Unfortunately, Python is really slow at this (compared to Ruby). Adding source tracking to the prog gen increased generation time for the prb1 flow from ~200ms to 4s! There may be some Pyo3 overhead, but I also noticed this was slow when implementing register description parsing from source file comments and that was a complete Python-side solution.

So the method I actually use to get the caller for the prog gen is this one: https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/pyapi/src/utility/caller.rs#L78-L90

You can see it is enabled via a STATUS attribute and this is to allow the user to enable/disable error backtracking by applying a --debug or -d switch to origen g.

Whenever the prog gen detects a frontend error during backend processing it will advise the user to do this.

On the backend, the node! macro has been enhanced to accept a special metadata argument as shown in this example (note the use of a ; to separate the (optional) metadata argument from the last node attribute:

https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/origen/src/prog_gen/flow_api.rs#L8-L11

Operation Identification

In Rust code the current operation can now be sniffed via the following method:

STATUS.operation()   // => Operation::Generate

Jinja2 Compiler and Compiler Tweaks

I moved the application templates from <app.root>/<app.name>/templates to <app.root>/templates. I've gone back and forth on which one makes sense, but let's settle on this new location. This follows the rationale that <app.root>/<app.name> is in the Python load path and is for Python code, non-Python code goes outside. The existing <app.root>/web directory already follows this convention and this brings templates into line.

I added an initial Jinga compiler, which at least worked for my use case: https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/python/origen/compiler.py#L279

I changed the default output directory from the compiler from output/renders/<template type>/ to simply ouput/compiled.

This was initially because I struggled to get the original to work with the new compiler, but also renders seems like a back end term that would be confusing to a user ("I ran the 'compile' command, why is it in 'renders'?") and grouping output by input template type doesn't seem to add much utility.

Differ Trait

I introduced a Differ trait - https://github.com/Origen-SDK/o2/blob/517de0c99782d8ab1ca7280f2f44eeacb7d884ed/rust/origen/src/utility/differ.rs#L7-L13

This is used in the relevant sections of the patgen and proggen and is to support future context-specific differs for certain files, e.g. a dedicated V93K flow file differ.

[ginty]

@coreyeng, I further enhanced the node! macro to allow children to be defined, makes writing processor test cases a bit easier:

https://github.com/Origen-SDK/o2/blob/f6cdb2f9fe5299e66249bf4d31eef0de82d504cc/rust/origen/src/prog_gen/processors/nest_on_result_nodes.rs#L94-L125

[ginty]

Just another general update: I changed the output directories to follow the convention that all dirs are lowercased and underscored, so output/V93KSMT7 => output/v93ksmt7

[coreyeng]

Looks good to me! Looking forward to trying to move our internal projects to using this stuff too. We've been all pattern-centric thus far. Unfortunately, we're kinda bogged down at the moment over here, but I'd recommend @chrisnappi just be aware of the current progress. At the very least we could probably take another shot at a scan flow generator or something to that effect.

I'll start playing with the "Python Source Filename/Line Number Tracking". That looks incredibly useful. Adding node children support is a nice enhancement too!

[ginty]

Thanks @coreyeng.

With regards to the source tracking, I have that up and running now in some of the prog gen code, so you can follow that for a usage example.

When generating nodes you should call the method src_caller_meta() to generate the meta-data input, as shown here:

https://github.com/Origen-SDK/o2/blob/ef0326aa460f3fa8b8067d63884962a3d695505a/rust/pyapi/src/prog_gen/interface.rs#L57

Then to use the backtracking I've added a trace! macro as show here:

https://github.com/Origen-SDK/o2/blob/ef0326aa460f3fa8b8067d63884962a3d695505a/rust/origen/src/prog_gen/processors/extract_to_model.rs#L57

The first arg is any operation that returns a Result and the second is a node that contains the info you want to trace. The macro will return another result that contains the original error plus the backtracking info (or advice on how to get it).

The macro calls this method which currently only does something when generating a program flow, but you can modify as required for how you want to handle it for pat gen: https://github.com/Origen-SDK/o2/blob/ef0326aa460f3fa8b8067d63884962a3d695505a/rust/origen/src/generator/node.rs#L70

Currently for prog gen it only adds the last caller from a flow file (that's what O1 did), however I'm now thinking that it might be better just to dump a full Python stack trace. It takes so long anyway that I doubt it will make much difference performance-wise, and its potentially much more useful for patgen where the error could originate in controller code as much as pattern source file code.

If you stick to using the above methods to consume this then we can convert them to do that when we get around to it.