Create IpPackager Testbench / Full-Featured-Example

[patrick-studer]

The packager thrives and prospers and we are adding many new features (also in the future with the new version).

Therefore, I would like to have a dummy hdl block (it can be only a simple top level file) that is packaged with every different version of IpPackager that exists (at the moment there is only the 2017_2_1 version).

This can be done by a fully-flagged package.tcl (maybe a better name that includes the version targeted IpPackage2017_2_1) that tests/shows every implemented feature by packaging the dummy hdl block.

I guess @jonasppsi you already have something like that, or how did you test new features before merging?

We could either place this in a testbench folder or use id as example (every feature used).

[jonasppsi]

So far I tested new features individually/manually in current IPs I have.

But you are right, it would help having all features summarized in a single IP Package.

I have a look into this.

[Botnic]

I'm not sure if having ALL features in the same test IP is a good idea. I fear it will be quite complicated to handle. I would prefer to have one "Test IP" for each feature. May be a little bit more work, but may be helpful when testing on different Versions.

[patrick-studer]

I'm not sure if having ALL features in the same test IP is a good idea. I fear it will be quite complicated to handle. I would prefer to have one "Test IP" for each feature. May be a little bit more work, but may be helpful when testing on different Versions.

@Botnic So all features from a single IpPackagerxxxx_y.tcl file must be working on the covered Vivado Versions (from the one starting at xxxx_y until the newest version.) If one "new" feature does not work with the actual IpPackager Version, a new version is needed. So having one full-featured example per feature-set (= per IpPackagerxxxx_y.tcl) may be a better solution.

Note: This image is just as reference and does not represent the actual agreed IpPackager Versions!

[Botnic]

@paedu92 I see you idea. How would you then test variants of a configuration? For example, there is the option to set the revision of the IP to a "string" or to "auto". But in one test file you will not be able to have both.

[patrick-studer]

@Botnic you are right. This exclusive (XOR) cases cannot be tested with a single Example/Test script. A further example where one test wouldn't be enough would be the "ALL" keywords that are present in some functions.

But if we look from a Vivado point-of-view, then it does not matter if the user sets a specific string or "auto". The IpPackager will call the same Vivado function to set the revision. The magic happens inside the IpPackager.tcl and is independent of the Vivado Version.

Further clarification

My intension with the Testbench/Example script is, that every User/Contributor/... has the chance to test features (new or existing) on an existing Dummy IP. So all of us test stuff with the same framework. At the moment, there is nothing common, everyone uses own IP's, for example from own projects.

The problem of this IPI Packaging thing is, that there is no possibility for self-checking testbenches. Sure, we can check if the script runs through to the end. But who will guarantee that the IPI-GUI looks correct. Or that the propagation of Parameters still works. This needs human interaction/inspection for sure...

So for me it makes no sense to create multiple feature testbenches, if the tester has to visually inspect all of them.

But if we have a common example script, it is easy to modify this (manually) and check different use cases. This is only necessary if someone fixes a bug or adds new functionality, or if someone reports a problem with an existing feature.

@Botnic do you agree with my explanation or would you still prefer single feature/function testscripts to really cover all configuration options?

[Botnic]

@paedu92 Thank you for your clarification

One way to check the result would be to compare the component.xml generated. I also like to look at the GUI. But to check if the script has done everything correctly it may be very difficult, specially as we are not looking at a IP that makes sense. Seeing some missing interface or other feature in the GUI could be very hard.

I agree that we should start with a single test IP trying to add as much features as possible. I just fear that this solution may not be sufficient in the future (but nothing will keep us from changing/adding if we have issues)

[patrick-studer]

@Botnic I see your point. But as you proposed, let us start first with a single file and if we see that extending the test-infrastructure is necessary it is always possible.