DirectML operator documentation inconsistencies

[gbionescu]

This is not actually an issue but more of a question or discussion.

I generated a bunch of JSON files to test all DML operators and found a series of issues and inconsistencies in the public documentation found here: https://docs.microsoft.com/en-us/windows/win32/api/directml

To generate accurate JSON files, I scraped the documentation pages and extracted information from the tables found at the bottom of each DML operator page. These JSON files have been tested using dxdispatch.

While extracting that information, I found a series of documentation inconsistencies (e.g. a few table headers contain upper characters as opposed to only lower characters or the documentation states that some DML operators accept specific inputs, but upon execution, DirectML complains that input parameters are invalid).

I don't have all of them noted down right now, but one quick example that I can point out is that in https://docs.microsoft.com/en-us/windows/win32/api/directml/ns-directml-dml_activation_celu_operator_desc the header contains "Supported Dimension Counts" while other pages reference the same header as "Supported dimension counts".

Another example is that DML_OPERATOR_ELEMENT_WISE_CLIP1 requires a field called MinMaxDataType, but upon execution, dxdispatch (or the DirectML dll) requires a field called ValueDataType. Adding a new field called ValueDataType works around that problem - note that both MinMaxDataType and ValueDataType are required.

I could add more examples, but I'd rather note down all of them separately.

1. Is the DirectML documentation publicly available? I'm thinking that I could create a PR for those pages that have minor inconsistencies.

2. Is this repo the right place to open issues on incomplete or invalid validation happening inside DirectML?

3. Is there a tool that I could use instead to generate those JSON files? I see that in dxdispatch https://github.com/microsoft/DirectML/blob/master/DxDispatch/tools/GenerateParsingHelpers.ps1#L3 there's a file referenced that (I assume) contains the schema for all operators. Is that open source? Or could it be made open source?

Thanks, Gabriel

[adtsai]

Hi, thanks for the feedback! The source that generates the DirectML documentation pages is publicly available - you can find it here. For corrections, feel free to send PRs directly to the MicrosoftDocs repo.

Is this repo the right place to open issues on incomplete or invalid validation happening inside DirectML?

Yes, if you think you've found a bug in our validation, please do file a GitHub issue and we'll look into it.

Specifically for CLIP1, here's the struct definition as it appears in our header DirectML.h:

struct DML_ELEMENT_WISE_CLIP1_OPERATOR_DESC
{
    const DML_TENSOR_DESC* InputTensor;
    const DML_TENSOR_DESC* OutputTensor;
    _Maybenull_ const DML_SCALE_BIAS* ScaleBias;
    DML_TENSOR_DATA_TYPE MinMaxDataType;
    DML_SCALAR_UNION Min;
    DML_SCALAR_UNION Max;
};

It appears that the documentation for this operator is correct, so this might just be a problem with the dxdispatch tool. @jstoecker FYI

Is there a tool that I could use instead to generate those JSON files?

Unfortunately we don't have a public machine-readable description of all our operators, but the closest thing is likely going to be the DirectML.h header itself. The header is going to be the canonical source of truth (and the docs should match the header - if they don't, let us know!)

You can find a copy of the header (and DLLs) inside our NuGet package. .nuget files are just zip files, so you should be able to rename the extension and extract the header if you'd like to take a look at it directly.

[jstoecker]

Another example is that DML_OPERATOR_ELEMENT_WISE_CLIP1 requires a field called MinMaxDataType, but upon execution, dxdispatch (or the DirectML dll) requires a field called ValueDataType. Adding a new field called ValueDataType works around that problem - note that both MinMaxDataType and ValueDataType are required.

Thanks, Gabriel, this is indeed a bug in dxdispatch. The generated parsing code for ELEMENT_WISE_CLIP1 is incorrect because the script you referenced is assuming all DML_SCALAR_UNION fields reference a second field named "ValueDataType".

I am in favor of publishing an open-source version of the DirectML API schema, but as Adrian mentioned we don't have anything that is suitable for public consumption right now. Hopefully that changes sooner than later, but for now Adrian's suggestions are good advice. It would be interesting to know more about your specific use case here to justify making a public stable schema.

Is the DirectML documentation publicly available? I'm thinking that I could create a PR for those pages that have minor inconsistencies.

The docs you see are produced from an open-source repo. You can open PRs in the MicrosoftDocs repo, but it's better to just open an issue on the DirectML GitHub repo (like this one) so we can fix it in both public and internal repos.

[gbionescu]

@adtsai thanks for the pointers, will take a look.

It would be interesting to know more about your specific use case here to justify making a public stable schema

@jstoecker I'm looking for a way of running DML operators (with a few specifics on what subsets of data to test) at the DirectML level and found dxdispatch quite useful, so I put something together to generate JSON files for inputs.

Do you have other suggestions on what I could use here? I assume that Microsoft has such tools, but probably most of them are closed source or not for public use :)

[jstoecker]

I'm looking for a way of running DML operators (with a few specifics on what subsets of data to test) at the DirectML level and found dxdispatch quite useful, so I put something together to generate JSON files for inputs.

That's good to hear! Just know that it's not going to be the most efficient (performance wise) if you end up chaining together operations into a graph. DxDispatch is really intended more for experimentation/debugging than as a foundational library for applications to build on. For a more general compute platform we expect most people are going through something like ONNX runtime, PyTorch, or TF: we have more sophisticated logic in these codebases for optimizing graph-level computations.

I know some others have expressed interest in a layer that is pure DirectX and has some of the sophisticated logic from our ML framework backends, but it doesn't exist as of today. Maybe one day!

Do you have other suggestions on what I could use here? I assume that Microsoft has such tools, but probably most of them are closed source or not for public use :)

Unfortunately, no. We do have some internal tests and tools as you'd expect, but no generalized tools suitable for public use. D3D12 is already a pretty low level for most developers to work with, so it's a fairly specialized group of users trying to call into DirectML APIs (hardware partners, game devs, ML frameworks).

Nevertheless, I think a public API schema has value. I will advocate for it but no promises we can publish anything anytime soon.