Compile shader at runtime

[amerkoleci]

Would be possible to compile shaders at runtime?

At the moment I saw there is the shaderc tool, but in scene graph there can be different light/shader permutation, how can this be achieved?

[andr3wmac]

It's not a proper implementation, but I accomplished this here: https://github.com/andr3wmac/Torque6/blob/master/lib/bgfx/tools/shaderc/shaderc.cpp#L679-L1807

Essentially I just changed main() into a function and started calling it from my engine. It's worked quite well for me.

[jpcy]

Watch out for fcpp closing stdout. Not very friendly behavior for a library.

[bkaradzic]

Fixed: https://github.com/bkaradzic/bgfx/commit/ac6d6bd9585576cd4075fad57794fe11ec836607

[shakesoda]

Is there any plan to support this other than via hacking shaderc?

[bkaradzic]

@shakesoda Yes: https://bkaradzic.github.io/bgfx/overview.html#todo

[bullno1]

That's a local link though. But bgfx doesn't even have file system access (and I like it that way), how will you deal with include file?

[bkaradzic]

Link is fixed. Yeah it doesn't have file access, and that's one of problems with it. My plan is to make it utility library (basically what @andr3wmac suggested above), and not part of the bgfx library. Compiling shaders in runtime doesn't make any sense in real-world, it's more for tools or hobbyist projects where prebuilding is hassle to them.

[slobodin]

Hello! Any progress on this issue?

[bensanmorris]

I also wanted to use shaderc as a library and in case it's of any use to anyone else this worked for me (in Visual Studio):

• Modify the shaderc tool's configuration type (shaderc's project properties / general tab) from executable to static lib and change the "Target Extension" to ".lib"
• If you compile it as is and then link it into your target executable along with the bgfx lib then you'll get a couple of errors indicating that getUniformTypeName(), nameToUniformTypeEnum() functions and the s_uniformTypeName array are already defined in bgfx lib (as they are also defined in the shaderc code).

To get around this...

• Mark getUniformTypeName() and nameToUniformTypeEnum() as extern, removing their definitions from the shaderc.cpp file and also remove the s_uniformTypeName array definition in the shaderc.cpp file
• Compile it to get a static lib version of the shaderc lib

You'll need to define compileShader() in the header file so you can call it and also expose methods to enable the caller to get at any compilation errors.

andr3wmac's solution to this is:

Add the following 3 function defintions to shaderc.h inside the bgfx namespace:

void compilerError(const char *_format, ...);
int compileShader(int _argc, const char* _argv[]);
void getShaderError(char* _outputText, uint16_t& _outputSize);
#define fprintf(target, format, ...) compilerError(format, ##__VA_ARGS__)

Modify shaderc.cpp by adding the definitions of the missing functions we declared above:

void compilerError(const char *_format, ...)
{
    va_list args;
    va_start(args, _format);
    _shaderErrorBufferPos += vsprintf(&_shaderErrorBuffer[_shaderErrorBufferPos], _format, args);
    va_end(args);
}

void getShaderError(char* _outputText, uint16_t& _outputSize)
{
    strcpy(_outputText, _shaderErrorBuffer);
    _outputSize = _shaderErrorBufferPos;
}

Modify compileShader() definition (and preceding lines) as follows:

char     _shaderErrorBuffer[UINT16_MAX];
uint16_t _shaderErrorBufferPos = 0;

int compileShader(int _argc, const char* _argv[])
{
    _shaderErrorBuffer[0] = '\0';
    _shaderErrorBufferPos = 0;

    bx::CommandLine cmdLine(_argc, _argv);

Compile it to get a shaderc lib which you can then link along with bgfx lib into your app.

[jarrettchisholm]

Hey @bkaradzic, if a PR was made for this kind of feature, would you consider it?

I ask, as I need runtime shader compilation (for faster development), and was thinking about implementing this. On your Kanban board I see you have a 'rewrite' of shaderc as a todo item, so I wasn't sure if you were planning to redo all of it or if that was related to making it a library.

[jarrettchisholm]

It looks like most of the code is implemented in C, with a little bit of C++ thrown in (i.e. std::string). While looking at the int compileShader(int _argc, const char* _argv[]) function, my thinking was to allow it to be called with something like std:istream and std:ostream for the input and output data. That way, if it's a library function call, the user can pass in the streams that correspond to the input and output (might not be a file), or if it's from the command line it could use the input and output streams and finally write the results to a file.

I was also thinking about passing around an std::ostream to use instead of stderr, so that a calling library could effectively buffer all of the 'log' output and possibly do something with it ( i.e. print to a log file , etc. ).

How do you feel about introducing some more C++ to the code? I'm not sure if there is a C way of doing this with the bx library ( there might be and I might just be missing it).

[bkaradzic]

bx uses Reader/Writer which can be anything: https://github.com/bkaradzic/bx/blob/master/include/bx/readerwriter.h

Anyhow, preferable way to do this is to create compileShader function above, that will call external process and compile shader: https://github.com/bkaradzic/bx/blob/master/include/bx/crtimpl.h#L93

[jarrettchisholm]

Hey @bkaradzic, thanks for the feedback!

I'm trying to do it by calling it as an external process, using the bx::ProcessReader class.

The code I'm running is:

    bx::Error error;
    auto processReader = bx::ProcessReader();
    if (!processReader.open("shadercRelease -f ../data/shaders/shader.fs.sc -o ../data/shaders/shader.fs.sc.bin --type f --platform windows --profile ps_4_0 -O 3")
    {
        throw std::exception("Unable to load shader file.");
    }
    else if (!error.isOk())
    {
        auto msg = std::string("Unable to load shader file: ") + error.getMessage().getPtr();
        throw std::exception(msg.c_str());
    }

    char buffer[2048];
    int32_t numCharactersWritten = processReader.read(buffer, 2048, &error);
    int32_t result = processReader.getExitCode();
    processReader.close();

    if (0 != result)
    {
    auto msg = std::string("Unable to compile shader file: ") + error.getMessage().getPtr();
    throw std::exception(msg.c_str());
    }

when I run my program, I get the following output:

ERROR: Failed to parse varying def file: "../data/shaders/varying.def.sc" No input/output semantics will be generated in the code!
Shader entry point 'void main()' is not found.
Failed to build shader.
An exception occured: Unable to compile shader file: ProcessReader: EOF.

The error message is coming from the bx::Error class, via error.getMessage().getPtr(). This makes sense, given that's what the ProcessReader sets in the bx::Error class.

However, if I try to get the resulting output from the actual command, I get this:

ERROR: Failed to parse varying def file: "../data/shaders/varying.def.sc" No input/output semantics will be generated in the code!
Shader entry point 'void main()' is not found.
Failed to build shader.
An exception occured: Unable to compile shader file: Ëþ.w,­©

I'm using auto msg = std::string("Unable to compile shader file: ") + buffer; to get the output from the actual command. But all I end up getting back is Ëþ.w,­©. Am I doing this right? I'm not sure I'm understanding exactly how the output from the command is getting read..

[jarrettchisholm]

If I append 2>&1 to the end of my command it seems to get some of the output data now! I guess I needed to redirect standard out to standard error.

It seems as though I'm getting some extra data at the end (i.e. on the last line I see a repeat of No input/output semantics will be generated in the corµ with the character µ at the end).

[jarrettchisholm]

Hmm...It's working more or less now, but if an error occurs, the exit code is 1999540969 (I'm on Windows 10 64bit).

If everything goes well, the exit code is 1392645460.

Anyone have this happen to them?

[jarrettchisholm]

Gah, this was happening because I was calling processReader.close() after processReader.getExitCode(). I moved the call to processReader.close() above processReader.getExitCode() and it works now :)

[bkaradzic]

auto processReader = bx::ProcessReader();

should be:

bx::ProcessReader processReader;

Less code, and not Crazy++11 style. :)

[jarrettchisholm]

Haha fair enough :)

[fredakilla]

for those looking for a library version. inspired from lumix engine shaderc hack (thanks nem0), I added some stuff to compile shaders in memory with this library version of shaderc : https://github.com/fredakilla/brtshaderc

main advantages are :

• it use original shaderc code and doesn't modify anything (just a wrapper and a project build)
• compiled shaders are not writen to disk.
• easy to manage defines at runtime.