USD Validation Framework

[tallytalwar]

Description of Proposal

Here is a proposal for core USD Validation Framework. This will replace existing UsdUtilsComplianceChecker and usdchecker utilities.

The rendered view of this document can be found here.

Contributing

• [x] I agree to and accept the Supplemental Terms.

[pixar-oss]

Looks good to me!

[hybridherbst]

Some notes after going through the document:

1. Currently, running a file through usdchecker and e.g. opening it with usdview logs entirely unrelated messages. Sometimes one is happy and then the other crashes, and vice versa. I think one defined goal should be that the errors, warnings and logs of usdchecker MyFile.usda are identical to the errors, warnings and logs from e.g. usdview MyFile.usda.

2. I would welcome a way to get structured output out of usdchecker. A tool with similar scope that does this pretty well is the glTF Validator, which has various options to e.g. output nicely usable JSON data.

3. Some of the current checks in usdview are neither Prim- nor Stage-related but need to check parts of the hierarchy (e.g. https://github.com/PixarAnimationStudios/OpenUSD/issues/2954, https://github.com/PixarAnimationStudios/OpenUSD/issues/2829, and others). How would these be represented?

4. At least currently, it seems the "best practice" way to actually validate a file is to run it through both usdchecker and trying to open it with usdview, and combining the log outputs. Similar to (1) I think a stated goal should be that asking one tool should be sufficient to understand the validitity of a file.

[tallytalwar]

Hi @hybridherbst , thank you for these notes and going over the proposal.

1. Some of these errors which usdview reports on the terminal are composition errors, since the composition errors will have their own validators in the new framework, these should be appropriately handled by the new usdchecker, and hence conform with what usdview report. That being said, if the error reported by usdview are not data conformance but more hydra related these won’t be reported by this new usdchecker, since the USD Validation project is catering to USD conformance and not Hydra.

2. Yes, I do plan to have some form of structured output with the new usdchecker.

3. A test writer has the option to write a validator rule which either operates on a given layer, or a stage or a prim. All these rule checks are catering to some form of USD spec / schema (either core USD spec and schemas, or client’s introduced schemas) conformance and not how hydra sees and interpolates this data. In the first example I believe a validator which checks topology of a usdGeom mesh should be able to handle it and get reported when usdchecker is run on it. That being said, any “Hydra Prim Error” will not be catered by the Usd Validation, as the framework is written on the usd side of things and not how hydra consumes it. I believe the second UsdSkel issue you quoted can be covered by writing a validator which takes a stage and does appropriate testing of the usdSkel data.

4. That is the goal to some extent, but again USD Validation will restrict to conformance of the USD spec/schema (core + client introduced) and not how Hydra consumes it.

I also want to mention that USD Validation will be more comprehensive to a specification and hence more expensive and explicitly run by the clients with an intended purpose to catch any errors / warnings which is making the asset non-conforming. On the other hand it is important for Hydra to quickly figure out any incorrect behavior as per its model and for rendering related task, so will do minimal checks, enough for it to answer this question.

[hybridherbst]

Thank you!

if the error reported by usdview are not data conformance but more hydra related these won’t be reported by this new usdchecker

As a developer I understand what you're saying on a separation-of-concerns level; however, as a USD user I'm not sure how I can make sense of that. To me that sounds like: usdchecker can give me the go for a file being perfectly valid, but then a perfectly written Hydra implementation could still give me different errors, and I still can't use the file.

To give a reference to how this is handled in the glTF world: if a file passes the glTF Validator Checks, it is expected to work in glTF viewers. If it doesn't then that's a bug in the respective glTF viewer. So as per the spec, a file that passes the validator MUST work in well-written and spec-conformant glTF viewers. This coupling of specifications ensures that the validator output can actually be trusted.

[tallytalwar]

Yup, In agreement. If all validators are robust, hydra shouldn't error or warn about anything. If it does, then either the validator needs to be updated or there is a bug in hydra, unless there is a gl driver bug or something.