Shader includes and named strings

[jeweg]

I'm trying to use shader includes, but need to use the fallback implementation regardless of what extensions the driver says it supports (nvidia nsight doesn't work at all if that extension is used). Easy enough: init(Shader::IncludeImplementation::Fallback).

But to then actually use the includes, I need to use named string. However, the named string implementation will just check for the ARB_shading_language_include extension and then use those functions. That defeats the purpose of choosing the fallback implementation above and nsight breaks.

Should/could there be a fallback implementation of named strings to go along with the one for shader includes? After all, these two pieces of functionality are tightly connected.

[scheibel]

You're right, Shading Language Include and Named Strings are tightly coupled and even defined and described in the same extension (https://www.opengl.org/registry/specs/ARB/shading_language_include.txt).

I had a look at the actual process that is triggered when compiling a globjects::Shader. Using the driver-supported implementation, the important steps are in order:

1. Shader::updateSource
2. ShadingLanguageIncludeImplementation_ARB::updateSources
3. source = shader->source()->string();
4. glShaderSource
5. Shader::compile
6. ShadingLanguageIncludeImplementation_ARB::compile
7. glCompileShaderIncludeARB

Using the fallback implementation (as configured using your example), the steps are the following:

1. Shader::updateSource
2. ShadingLanguageIncludeImplementation_Fallback::updateSources
3. resolvedSource = IncludeProcessor::resolveIncludes(shader->source(), shader->includePaths());
4. glShaderSource
5. Shader::compile
6. ShadingLanguageIncludeImplementation_Fallback::compile
7. glCompileShader

This should mean that the actual compiled shader source has all includes inlined (resolved), disregarding the availability of a named string registry in the driver. What I'm currently not sure of (and I suppose it actually may happen) is that globjects uses the driver named strings to store them and read them back every time the shader gets compiled. This should be changed.

Do you experience an actual problem or is this issue because of your code review? Either way, you're right and the globjects::NamedString implementation shouldn't be misleading.

[andrew-kennedy]

I've been poring over all the code I can find related to shader includes and even read over the whole ARB extension spec, and I'm still kind of at a loss as to how to use the extension with globjects on an Intel Iris GPU with OpenGL 4.1. I cannot get the include to find my file, it always errors with #warning: Did not find include useful_functions.glsl. I have the absolute path where the file is located set in the the include paths, so I must be missing something else basic. However, the documentation surrounding shader includes seems to be minimal and I didn't see any use of includes in the example projects.

[scheibel]

Can you provide the source code that results in this error?

We use the shading language includes this way:

Registering named strings https://github.com/cginternals/gloperate/blob/51912a7042ee5815ad19c543f14da55b0e3bb4c7/source/gloperate/source/stages/demos/LightTestStage.cpp#L172

Including a registered named string https://github.com/cginternals/gloperate/blob/master/data/gloperate/shaders/lightProcessing.glsl

We don't use a specialized interface to compile programs and shaders, we rely on the automatic strategy selection by globjects.

[andrew-kennedy]

I'm a bit confused why shaders take include paths if the correct way to use include paths are through named strings?

I was previously using this constructor Shader::Shader(const GLenum type, AbstractStringSource * source, const IncludePaths & includePaths) with a std::vector<string> of paths passed into the include paths parameter as I thought that was the purpose it served.

[scheibel]

The include paths are specified by OpenGL as well. If you write a shader that includes another shader via a named string, this named string has not to be fully qualified. If it starts with a /, it is assumed to be fully qualified and a lookup can be directly issued. If it doesnt start with a / the driver (and our emulation) iterates over all given include paths, prepend them to the named string identifier and checks if such a named string exists. If it exists, this one is used.

[andrew-kennedy]

Ahh, okay. That makes a lot of sense. So in principle if I have a shader located inside my program's working directory at the path ./resources/shaping_functions.glsl and the fully qualified path is /Users/andrewkennedy/code/resources/shaping_functions.glsl I would set the Include path to /Users/andrewkennedy/code/resources and a named string of vertex_shader.glsl and then when including that shader via its named string I could just write #include <shaping_functions.glsl>?

[scheibel]

It's a bit more complicated than that. But in general, we have helper functions which mirror a file system for the use in GLSL shaders.

The includes you may use in a GLSL shader doesn't directly come from the file system (I think a security feature?); instead you have to register named strings that follow a path-like scheme and that reference source code that is directly inserted into the shader source when includeded. These source code snippets may come from files (see example above) but doesn't have to. You tell the OpenGL driver via C-strings which include path will resolve to which source code snippet. If you mirror your file system like we do, then it seems as if you're including files like in C or C++.

[andrew-kennedy]

Yeah I read through the named string spec enough that I understand that OpenGL doesn't have filesystem access and that using named strings to create a "mirror" of the filesystem is necessary for true C-style includes, I just wondered if the method I described was how globjects did that mirroring.

I'm more just trying to understand how to properly use the many features of globjects. I got includes to work using fully qualified named strings like you did here but wondered what the proper way to use named strings and include paths would be in order to get behavior close to C-style includes, where I could put all my shader files in one directory and easily break them into separate files and include them in each other for composability.

[scheibel]

Yeah we do exactly that using the source code snipped I linked to. And so far I can recommend this. During the program initialization, we start a recursive file search within our data directory, search for all .glsl files and register them as named strings, where we crop the path before the data folder to provide compatibility for other locations and deployments.

Example source:

/home/scheibel/myproject/data/shaders/shading/phong.glsl
/home/scheibel/myproject/data/shaders/geometry/normals.glsl

Example named strings:

/data/shaders/shading/phong.glsl
/data/shaders/geometry/normals.glsl

You may even crop the /data and /shaders as well.