Integrating "gaussian blurs for GLSL" to OES Fragment Shader

[konidk]

Hi,

I'm currently working on a project that requires reproducing the human eyesight using the camera passthrough and then add a blur filter to it in order to reproduce myopia. I need to be able to adjust the blur degree (from mild to important level of myopia).

The thing I've done is modifying the fragment shader from OES Shader (wich is used by GVRCameraSceneObject) by using the GLSL code found here (at the bottom of the webpage). Here is the new fragment shader I've got (with the great help of @NolaDonato) :

        String FRAGMENT_SHADER =
                "#extension GL_OES_EGL_image_external : require\n"
                        + "precision mediump float;\n"
                        + "uniform samplerExternalOES main_texture;\n"
                        + "uniform float blur_threshold;\n"
                        + "in vec2 vTextureCoord;\n"
                        + "out vec4 outColor;\n"
                        + "vec3 mosaic(vec2 position) {\n"
                        + "    vec2 p = floor(position) / blur_threshold;\n"
                        + "    return texture(main_texture, p).rgb;\n"
                        + "}\n"
                        + "vec2 sw(vec2 p) {return vec2( floor(p.x) , floor(p.y) );}\n"
                        + "vec2 se(vec2 p) {return vec2( ceil(p.x) , floor(p.y) );}\n"
                        + "vec2 nw(vec2 p) {return vec2( floor(p.x) , ceil(p.y) );}\n"
                        + "vec2 ne(vec2 p) {return vec2( ceil(p.x) , ceil(p.y) );}\n"
                        + "vec3 blur(vec2 p) {\n"
                        + "vec2 inter = smoothstep(0., 1., fract(p));\n"
                        + "vec3 s = mix(mosaic(sw(p)), mosaic(se(p)), inter.x);\n"
                        + "vec3 n = mix(mosaic(nw(p)), mosaic(ne(p)), inter.x);\n"
                        + "return mix(s, n, inter.y);\n"
                        + "}\n"
                        + "void main() {\n"
                        + "  outColor = vec4(blur(vTextureCoord *blur_threshold), 1.0);\n"
                        + "}\n";
        setSegment("FragmentTemplate", FRAGMENT_SHADER);

I use the blur_threshold variable in order to adjust the blur intensity. The problem is that this is definitely not the best way to have a blurry vision, this is just moving around pixels in a way that the image gets distorted but it doesn't reproduce a real blur.

I have just found this new filter called gaussian blurs for GLSL which seems to be a best solution for my problem, but as long as I don't know anything about GLSL I can't get it to work at all. I tried adapting the OES Fragment Shader to implement this code with no success. Any help would be greatly appreciated, thanks :)

[NolaDonato]

Thanks so much for your project! My left eye cannot see distance at all but my right eye can so if you are successful you will make VR more comfortable for me. I will look at your filter and see if I can get it to work in a test app.

[NolaDonato]

The two pass method requires you to render the same mesh twice with different shaders. While this is possible in GearVRF, you must render into an external texture and it is somewhat messy. I have implemented a single-pass Gaussian blur which is less efficient than the two pass one but easier to code. Here is the shader:

extension GL_OES_EGL_image_external_essl3 : require

precision mediump float; uniform float u_resolution; uniform samplerExternalOES main_texture;

in vec2 vTextureCoord; out vec4 FragmentColor; float normpdf(in float x, in float sigma) { return 0.39894 exp(-0.5 x x / (sigma sigma)) / sigma; } vec4 blur(vec2 texCoord, vec2 resolution) { const int size = 11; const int kSize = (size - 1) / 2; float kernel[size]; vec3 final_colour = vec3(0.0);

//create the 1-D kernel
float sigma = 7.0;
float Z = 0.0;
for (int j = 0; j <= kSize; ++j)
{
    kernel[kSize + j] = kernel[kSize - j] = normpdf(float(j), sigma);
}

//get the normalization factor (as the gaussian has been clamped)
for (int j = 0; j < size; ++j)
{
    Z += kernel[j];
}
//read out the texels
for (int i = -kSize; i <= kSize; ++i)
{
    for (int j = -kSize; j <= kSize; ++j)
    {
        final_colour += kernel[kSize + j] * kernel[kSize + i] * texture(main_texture, texCoord + vec2(float(i), float(j)) / resolution).rgb;
    }
}
return vec4(final_colour / (Z * Z), 1.0);

} void main() { vec2 resolution = vec2(u_resolution, u_resolution); FragmentColor = blur(vTextureCoord, resolution); }

To use the shader you have to set up the resolution as a uniform in the BlurShader class and then give it a value in your material. In the BlurShader I put the fragment and vertex shader sources into file which go in the res/raw directory.

public class BlurShader extends GVRShaderTemplate { private static String fragTemplate; private static String vtxTemplate;

public BlurShader(GVRContext context)
{
    super("float u_resolution", 300);
    fragTemplate = TextFile.readTextFile(context.getContext(), R.raw.gaussianblur);
    vtxTemplate = TextFile.readTextFile(context.getContext(), R.raw.vertex);

    setSegment("VertexTemplate", vtxTemplate);
    setSegment("FragmentTemplate", fragTemplate);
}

}

Here is how you set up the material:

        GVRMaterial cameraMtl = cameraObject.getRenderData().getMaterial();
        GVRSceneObject blurryCamera = new GVRSceneObject(context, context.createQuad(3.6f, 2.0f), cameraMtl.getMainTexture(), GVRMaterial.GVRShaderType.OES.ID);
        GVRRenderData rdata = blurryCamera.getRenderData();
        GVRMaterial blurMtl = rdata.getMaterial();

        blurMtl.setFloat("u_resolution", 1024.0f);
        blurMtl.setTexture("main_texture", cameraMtl.getMainTexture());
        rdata.setShaderTemplate(BlurShader.class);
        rdata.bindShader(scene);

[konidk]

The project is about helping to understand how visual impairments impact lives of visually impaired people. This should help the research in visual research ! We are working with an engineering school laboratory. I will give you more information once it's finished :)

The fragment shader you provided reproduce perfectly myopia, it's exactly what I'm searching for. This is a realistic blurry vision. Thanks a lot ! But I still can't use it because the calculation takes too much time and it makes the camera laggy on my Galaxy S7 (should do the same thing on your Galaxy S7 Edge ?). I tried using the camera with 30 fps mode using cameraObject.setUpCameraForVrMode(0); but the problem persists. It feels so frustrating because the blurry vision is really good, hope we can improve the calculation in some way.

[NolaDonato]

The two pass method may be faster because gaussian blur is a linearly separable operation. Mihail is working on improving the render to texture feature which is what is needed to do a two pass solution. I will look at implementing that in GearVRF but it might take a few days because we are working on getting out the next release.

[konidk]

I understand, please keep me updated about this new feature. Keep up the good work ! I will stay tuned.

[konidk]

If this can help, this link shows how to divide the process into two passes.

[NolaDonato]

I worked on your issue this weekend. I grabbed PR #1028 that Mihail is working on and tried to use it to render a blurred version of the camera input to a texture and subsequently use that texture on another object. This is the only way I can get multiple passes to work - if I render to a texture. Unfortunately, when I tried this approach the resulting quad is black. Mihail is on PTO for a few days, I will work with him when I get back to debug why the new render-to-texture code doesn't work for our case.

[konidk]

Ok, please keep me updated. Good luck.

[NolaDonato]

I have an update for you. Combining Mihail's pull request #1028 (render to texture) and my new pull request #1078 (authoring post effect shaders) I got your gaussian blur to work with two render passes to an external texture which I then mapped to a quad. We are still fine tuning the APIs and have not merged these pull requests to the master branch yet. I expect the APIs are not solid and will change. If you feel extremely adventurous you can build the GearVR framework yourself, apply these two patches and I will send you a working version of the gaussian blur sample Mihail and I worked on in PR #1028.

[konidk]

Glad to read this ! Hope the performance issue will be resolved using this method. The project I work on will reach its end on Monday 27 of March, it does leave us time to wait for the APIs to be updated.

However, I have to give back a first version of the project with a working blur vision by Friday, do you think that the APIs will be updated by this time ? Otherwise I thought of implementing the "mean filter" or the "median filter" as easier alternatives to blur the vision (even if it's not as accurate as the gaussian filter). In this case, I will also need your help for the GLSL code (could be quick seeing how simple these two filters are).

[NolaDonato]

I have attached a working sample of the two pass gaussian blur. This sample will work with a GearVRF built with the two patches. Here is how you get that:

1. Clone the GearVRF git repository - master branch into a local repo: git clone https://github.com/Samsung/GearVRf git checkout -b rendertotexture
2. Apply the first patch #1028 git pull git://github.com/liaxim/GearVRf.git rendertotexture
3. Apply the second patch #1078 git pull https://github.com/NolaDonato/GearVRf.git posteffect
4. Build your version of GearVRf using Android Studio Build Instructions: https://resources.samsungdevelopers.com/Gear_VR/020_GearVR_Framework_Project/060_Gear_VRf_Developer_Guide/110_Build_Instructions#Building_and_Running_GearVRf_Applications
5. Build your application using this version of GearVRF Make sure you have this line in gradle.properties (it will make the sample use your built version of GearVRf instead of the current release). useLocalDependencies=true

Directory structure should look like: root/GearVRf/GVRf/Framework... root/GearVRf-Demos/Test... (the sample app)

test.zip

[NolaDonato]

I have an update. I worked on render to texture and have an official pull request for this feature which will be in release 3.2. Patch #1099 and #1098 support what you are trying to do - render a two pass post effect into a texture. Pull request #424 from GearVRf-Demos is a sample implementing gaussian blur by rendering into a texture. To increase performance you could make the GVRRenderTexture 512x512 instead of 1024x1024.

Clone the GearVRF git repository - master branch into a local repo:
git clone https://github.com/Samsung/GearVRf
git checkout -b rendertotexture
Apply the first patch #1098 
git pull git://github.com/NolaDonato/GearVRf.git posteffect3
Apply the second patch #1099
git pull https://github.com/NolaDonato/GearVRf.git shadowmap_fix
Build your version of GearVRf using Android Studio
Build Instructions: https://resources.samsungdevelopers.com/Gear_VR/020_GearVR_Framework_Project/060_Gear_VRf_Developer_Guide/110_Build_Instructions#Building_and_Running_GearVRf_Applications
Build your application using this version of GearVRF
Make sure you have this line in gradle.properties (it will make the sample use your built version of GearVRf instead of the current release).
useLocalDependencies=true

I have attached the sample to this issue but it is PR #424 in the gearvrf/GearVRf-Demos repo.

gvr-blurfilter.zip

[konidk]

Thank you so much. I totally forgot to answer your precedent reply. Well, I got it to work following your first intructions, but I'll definitely try the new ones.

EDIT : Just tried your test app "gvr-blurfilter.zip", it works like a charm ! Really smooth rendering, impressive. Can't wait to implement this to my main app. I'll keep you updated :)

[liaxim]

@konidk Sounds like it is all good. Planning to close the issue if we don't hear from you.