Probem Determining Exported Symbols and Versions from cl.xml

[Kerilk]

I am trying to generate the icd_exports.map of the ICD loader from the cl.xml file. For now it seems impossible without substantial amount of hacks:

• gather commands present in the loader:
  • gather commands.command nodes,
  • filtering vendor specific functions,
  • filter extensions functions that are not/have never been entry points in the loader:
    • clIcdGetPlatformIDsKHR
    • clCreateProgramWithILKHR
    • clTerminateContextKHR
    • clCreateCommandQueueWithPropertiesKHR
    • clEnqueueMigrateMemObjectEXT
• find commands original OpenCL versions
  • for non extensions, this can be obtained from the feature node version and their require.command nodes
  • for non-deprecated extension functions, this can be obtained by scanning the corresponding commands.command node suffix
  • you are left with clCreateFromGLTexture2D and clCreateFromGLTexture3D witch have CL_EXT_SUFFIX__VERSION_1_1_DEPRECATED, so the scan assigns them as 1.1 whereas they are 1.0 functions

This seems a bit too convoluted. Nonetheless this process would be necessary to automatically generate a mock-up/intercept library, or bindings.

Several possibilities exist to alleviate the problem, here are some examples:

• For presence in the loader:
  • add an attribute telling which function are entry points for the icd loader,
• For version of the API either
  • add an attribute telling in which OpenCL version the symbols appeared. This would be redundant with some of the comments, but maybe that is not so much of an issue?
  • add version information for all extensions (cl_khr_d3d11_sharing description in the xml doesn't state it appeared in 1.2)

I am unsure what the best course of actions would be here, or if I missed something obvious. Any thoughts on the matter?

Thanks,

Brice

[oddhack]

Some feedback, speaking as the person who wrote cl.xml, the XML schema, and most of the Python code that operates on it. TL;DR I can probably assist with this and am probably the best person to do so, although I do need to plan it around other Khronos responsibilities so your timescale is important to know if I'm going to help. Also, I'm unclear if you are doing this as an officially Khronos-sponsored activity, or is this an external project?

cl.xml is (or at least, should be) a complete description of the APIs, but the only current use case is generating the code fragment includes for the specs. I'd like to generate C headers too, but without a clear demand for that from the working group, it has been a questionable use of my time. cl.xml obviously wasn't created with the loader in mind, but I expect that some relatively minor amount of additional tagging or schema additions could accomodate those needs.

While you can certainly write your own XML processing code, which I guess? you've been trying to do based on the issue text, I think you would be better off creating a new generator script that plugs into the existing Python framework. CL doesn't do that yet, but Vulkan and OpenXR both have a lot of custom generators used by the loader and validation layers, CTS, and other ecosystem components. The framework operates by being given a set of core versions & extensions to process, traversing their dependency graph through cl.xml to determine the corresponding tokens, structures, and commands, and calling back to the generator to emit those in whatever fashion is needed.

Without knowing how the ICD determines the supported commands I can't be precise about this, but it sounds like it would be straightforward to fit into the generator framework, either with an explicit exclusion list of commands that you list above, or additional XML attributes as you note.

Determining the API <-> version/extension mapping is not conceptually hard, although it would need some additions to the Python codebase to propagate that information down from the \ / \ tags to the individual APIs. We have a potential use for this in Vulkan as well, so it's been sitting on my queue for a while. The interesting, and rare (for anything except base API types) case is when multiple core versions or extensions require the same API, and the simplest thing to do is probably just track all of them and let the generator script decide amongst them.

I don't know if you need the deprecation information for the ICD? I left this with adding attributes containing the deprecation macros, but again, lacking a requirement for autogenerated headers, this is probably a bit incomplete and/or out of date. Also the particular syntax ('prefix'/'suffix' attributes on \ tags) was chosen solely to have an place to record this information, and I'm not attached to it. A better approach might be to capture this information in the version \ tags and propagate it to the corresponding commands from there. I was literally just reflecting the headers as written today since I had no immediate use for the information.

Lastly, I and other people working a lot with the schema and framework inside Khronos are trying to keep them as unified as possible across the four or five APIs that use them. So if you do end up needing schema and/or scripting changes to do this, I want to be in the loop on it.

[Kerilk]

First, thanks for the very detailed answer.

For now let's say this is a side project, but as maintainer of ocl-icd, and Ruby opencl bindings, and member of the Khronos OpenCL working group, I see an interest in the issue being discussed. I am really not an expert in XML schema design so I won't be pressing for specific solutions.

For more precise answers to your questions:

• Time frame is not important, I have been getting by generating ocl-icd for years using a simple yaml file containing the public APIs (with their original version tokens), and can certainly use the same strategy for current projects. It just seemed cleaner to use the xml definitions.
• Deprecation information is not useful for the icd, only original symbol presence, signature, and versioning matter.
• For bindings, version information and deprecation information are useful. Presence in the loader is also very important for automatically generated bindings because you want to know which symbol you can access directly from the library file and which you need to query through clGetExtensionFunctionAddressForPlatform.
• I'll take a look at the Python framework and it's uses in the Vulkan loader and try to see how the OpenCL loader would benefit from using it.

I do like your idea of capturing the original version, deprecation version in the tag, and maybe keep the prefix and suffix as boolean attributes for commands (allowing to regenerate identical headers). For now, the coherency of the headers on this account is questionable as some symbols have neither prefix nor suffix (there might be a good reason for that, or not).

[bashbaug]

FWIW, I've been experimenting a lot with code generation using the cl.xml file recently and things are generally going fairly well. I've focused mostly on core APIs for now, but I'd like to experiment with auto-generating an extension loader from the XML file at some point in the future.

I looked at the existing Python framework but it looked like overkill for what I needed, so I ended up using a Python + Mako toolchain instead, which was sufficient for my needs.

Don't take it too seriously, but in case it's useful, an example can be found here.

Regarding the ICD loader exports specifically, I don't think that the ICD loader should export any extension functions. See:

https://github.com/KhronosGroup/OpenCL-ICD-Loader/blob/master/loader/linux/icd_exports.map

A few extension functions are implemented in the ICD loader, but my understanding is that these are for backwards compatibility only, and that future extension APIs will not modify the ICD loader. See:

https://github.com/KhronosGroup/OpenCL-ICD-Loader/blob/master/loader/icd_dispatch.c#L1585