This PR adds the Shadereval tasks, Shadereval looks at "creative coding" by comparing models on shadercode(glsl with Shadertoy.com syntax). There is (currently) two tasks:
shadereval-1: ReturnCompletion as seen in demo space. Using a subset of Shadertoys-fine dataset to predict the return statement of functions. Metric used is exact_match. The original implementation is hosted as an EvaluationSuite inside the demo space.
shadereval-2: FunctionCompletion as seen in demo space, Using a processed version of the Shadertoys dataset, taking docstrings immediately before a function as a prompt and combines that with the function header as input. Evaluated with a custom metric (hosted in the demo space) that first compares the text statically, and then renders frames of prediction and reference.
The paper for the first task and dataset has been written but not not be published - it can be omitted from this PR since it was more of a prototype.
The second task is nearly done as part of my Thesis, we hope to publish this eventually too. The implementation should be done by October. The dataset is essentially locked but will be renamed. This PR is our reference implementation.
Remaining tasks:
[x] add documentation (including results)
[ ] cleanup various comments, especially #TODO:...
revival of #97
This PR adds the Shadereval tasks, Shadereval looks at "creative coding" by comparing models on shadercode(glsl with Shadertoy.com syntax). There is (currently) two tasks:
shadereval-1: ReturnCompletion as seen in demo space. Using a subset of Shadertoys-fine dataset to predict the return statement of functions. Metric used is
exact_match. The original implementation is hosted as anEvaluationSuiteinside the demo space.shadereval-2: FunctionCompletion as seen in demo space, Using a processed version of the Shadertoys dataset, taking docstrings immediately before a function as a prompt and combines that with the function header as input. Evaluated with a custom metric (hosted in the demo space) that first compares the text statically, and then renders frames of prediction and reference.
The paper for the first task and dataset has been written but not not be published - it can be omitted from this PR since it was more of a prototype. The second task is nearly done as part of my Thesis, we hope to publish this eventually too. The implementation should be done by October. The dataset is essentially locked but will be renamed. This PR is our reference implementation.
Remaining tasks:
#TODO:...