three and readRenderTargetPixels

[ua4192]

Hi.

I am trying to use gl.readPixels functionality in order to retrive information from GPU side. Basically I want to retrieve the position and vertex id using this function. I am cuerrently using THREEJS abstraction function called readRenderTargetPixels

function pick(){
    var fb = gl.createFramebuffer();
    console.log(canvas_h, canvas_w, mouse);
    pickingRenderTarget = new THREE.WebGLRenderTarget( canvas_w, canvas_h, { autoClear: true, minFilter: THREE.LinearFilter, magFilter: THREE.NearestFilter, format: THREE.RGBAFormat, type: THREE.FloatType } );
    var read = new Float32Array( 4 );
    renderer.render( scene, camera, pickingRenderTarget );
    console.log(pickingRenderTarget.height);
    renderer.readRenderTargetPixels( pickingRenderTarget, 10, 10, 1, 1, read );
    console.log(read);
    var v = new THREE.Vector4(read[ 0 ], read[ 1 ], read[ 2 ], read[3]);
        retrun v;
}

My problem is that when I try to use this function, which I have used in standard bufferGeomtries in THREEJS, I get this error:

threejs.js:25333 THREE.WebGLRenderer.readRenderTargetPixels: readPixels from renderTarget failed. Framebuffer not complete.

I am starting to think that this issue is linked with the internal WEBGL behabiour of Nexus.

Is there any way to achieve this?

[ponchio]

Ciao, framebuffer complete seems related to the framebuffer being allocated by the graphic card, and you make a read just after having created the target.

See here: https://learnopengl.com/Advanced-OpenGL/Framebuffers

You might want to use this code fragment to check for completion (not tested!

if(gl.CheckFramebufferStatus(gl.FRAMEBUFFER) == gl.FRAMEBUFFER_COMPLETE)

Additional notes: 1) if you are picking a point, you don't really need to render the canvas_wxcanvas_h pixels, you need just one. 2) Do you really need to create the buffer every pick? Create one at the beginning and reuse it. 3) The id attribute would be interpolated in the pixel shader, how do you plan to solve this? 4) If you only need 4 different id you could use rgba attribute which interpolate each channel separately and have 4 possible overlapping regions.

All of this provided I manage to support attributes in Nexus.

On Wed, Oct 30, 2019 at 11:59 AM Alex notifications@github.com wrote:

Hi.

I am trying to use gl.readPixels functionality in order to retrive information from GPU side. Basically I want to retrieve the position and vertex id using this function. I am cuerrently using THREEJS abstraction function called readRenderTargetPixels

function pick(){ var fb = gl.createFramebuffer(); console.log(canvas_h, canvas_w, mouse); pickingRenderTarget = new THREE.WebGLRenderTarget( canvas_w, canvas_h, { autoClear: true, minFilter: THREE.LinearFilter, magFilter: THREE.NearestFilter, format: THREE.RGBAFormat, type: THREE.FloatType } ); var read = new Float32Array( 4 ); renderer.render( scene, camera, pickingRenderTarget ); console.log(pickingRenderTarget.height); renderer.readRenderTargetPixels( pickingRenderTarget, 10, 10, 1, 1, read ); console.log(read); var v = new THREE.Vector4(read[ 0 ], read[ 1 ], read[ 2 ], read[3]); retrun v; }

My problem is that when I try to use this function, which I have used in standard bufferGeomtries in THREEJS, I get this error:

threejs.js:25333 THREE.WebGLRenderer.readRenderTargetPixels: readPixels from renderTarget failed. Framebuffer not complete.

I am starting to think that this issue is linked with the internal WEBGL behabiour of Nexus.

Is there any way to achieve this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/57?email_source=notifications&email_token=AAUDV25REUNXO6QTHCMN5TDQRFSIJA5CNFSM4JGXIXMKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HVLYJSA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUDV26AJWPJRCO42KWPJI3QRFSIJANCNFSM4JGXIXMA .

[ua4192]

Hi.

Many thanks, my problem was more evident, my problem was due to the fact the canvas_h value was not yet set when I defined webglRenderTarget. Now it works as expected.

: )

I got to get the position from GPU by readRenderTargetPixels.

I don't know id I need to create another ticket, but now my problem is that I don't know how to use uv geometry attribute with nexus library and threejs.

I tries to decompose the 3 floats of color geometry attribute, in order to use 2 first floats for color (RGB) and third for vertex id, but I realized that color attribute is normalized when nxz is built.

Therefore I need to know how to use uv because if I have properly understood this attribute is not normalized.

My idea is to use 1 of uv components in order to identfy vertex id.

I have tried to use it like color but uv is always set to 0 when I load the nxz model, maybe I guess because I don't know how to use it.

I have gott to use a custom RawShaderMaterial with Nexus but until I won't be able to use uv I won't be able to achive what I need.

Here my shader in order to give you some hints. By the way nexus works with webg2, what is wonderfull. :)

Your product is really amazing !!!!

<script id="fragmentShader" type="x-shader/x-fragment">
    precision highp float;
    in vec3 vColor;
    in vec2 vUv;
    out vec4 myColor;
    void main() {
        #ifdef PICKING
            myColor = vec4(vColor, vUv[0] + 1.0);
        #else
            myColor = vec4(vColor, 1.0);
        #endif
    }
</script>

<script id="vertexShader" type="x-shader/x-vertex">
    precision highp float;
    in vec3 color;
    in vec3 position;
    in vec2 uv;
    uniform vertexData {       
        mat4 projectionMatrix;     
        mat4 viewMatrix;           
        lowp vec4 selectionColor;  
        lowp vec3 lPosition;       
        lowp vec3 ambientColor;    
        lowp vec3 diffuseColor;    
        lowp vec3 specularColor;   
        lowp float shininess;      
    };                             
    out vec2 vUv;
    out vec3 vColor;                           

    void main() {
        vUv = uv;
        gl_Position = projectionMatrix * viewMatrix * vec4( position, 1.0 );
        #ifdef PICKING
            vColor = gl_Position.xyz;
        #else
            vColor = color.xyz;
        #endif 
    }

Here how I define materials for picking Scene and for scene:

var vertexUniformBlock = new THREE.UniformBlock();
vertexUniformBlock.setName( 'vertexData' );
vertexUniformBlock.add( { value: camera.projectionMatrix } ); // projection matrix
vertexUniformBlock.add( { value: camera.matrixWorldInverse } ); // view matrix
vertexUniformBlock.add( { value: new THREE.Vector4(1.0, 1.0, 0.0, 0.5) } ); // selectionColor
vertexUniformBlock.add( { value: camera.position } ); // light position
vertexUniformBlock.add( { value: new THREE.Color(0x333333) } );
vertexUniformBlock.add( { value: new THREE.Color(0x333333) } ); // ambient color
vertexUniformBlock.add( { value: new THREE.Color(0xaaaaaa) } ); // diffuse color
vertexUniformBlock.add( { value: new THREE.Color(0xcccccc) } ); // specular color
vertexUniformBlock.add( { value: 4 } ); // shininess

var material = new THREE.RawShaderMaterial( {
    uniforms: { "time": { value: 0.0 } },
    uniformBlocks: [ vertexUniformBlock ],
    vertexShader: "#version 300 es\n" + document.getElementById( 'vertexShader' ).textContent,
    fragmentShader: "#version 300 es\n" + document.getElementById( 'fragmentShader' ).textContent,
    transparent: true,
    side: THREE.DoubleSide
} );

var pmaterial = new THREE.RawShaderMaterial( {
    uniforms: { "time": { value: 0.0 } },
    vertexShader: "#version 300 es\n#define PICKING\n" + document.getElementById( 'vertexShader' ).textContent,
    fragmentShader: "#version 300 es\n#define PICKING\n" + document.getElementById( 'fragmentShader' ).textContent,
    uniformBlocks: [ vertexUniformBlock ],
    side: THREE.DoubleSide
} );

This is how I deal with pickingScene and scene when loading model:

function onNexusLoad() {
    sceneBox.union(nexus_obj.geometry.boundingBox)
    var geometry =  nexus_obj.geometry;
    redraw = true;
    nexus_obj.material = material;
    scene.name  = "scene";
    scene.add(nexus_obj);
    var pnexus_obj = new THREE.Mesh(nexus_obj.geometry);
    pnexus_obj.autoMaterial  = nexus_obj.autoMaterial;
    pnexus_obj.onAfterRender = nexus_obj.onAfterRender;
    pnexus_obj.material = pmaterial;
    pickingScene.add(pnexus_obj);
    pickingScene.name  = "pickingScene";
}

And here my picking function:

function pick(){
    var fb = gl.createFramebuffer();
    var read = new Float32Array(4);
    console.log(canvas_w, canvas_h);
    pickingRenderTarget = new THREE.WebGLRenderTarget( canvas_w, canvas_h, { autoClear: true, minFilter: THREE.LinearFilter, magFilter: THREE.NearestFilter, format: THREE.RGBAFormat, type: THREE.FloatType } );   
    renderer.render( pickingScene, camera, pickingRenderTarget );
    renderer.readRenderTargetPixels( pickingRenderTarget, mouse.x, canvas_h - mouse.y, 1, 1, read );
    console.log(read);
}

Please help me........

I am really interested in using your solution but being able to identifying vertices is a must.

Best regards

[ua4192]

Hi again.

I am guessing if this could simplify the problem you described about merging vertices and dealing with vertex_id.

Would it make more simple if the id was by mesh.

I mean. If I have a obj file with multiple meshes, for example 10 meshes, would it be possible at least to identify this 10 meshes with 10 different ids (0-9)?

The goal for me is to build a nxz file composed of multiple meshes coming from obj file but somehow being able to identify each mesh inside the nxz once loaded using threejs.

Best regards

[ua4192]

Any feedback about this topic? Best regards

[ponchio]

Hi, sorry for the delay (conferences and deadlines, in the meanwhile).

The easiest solution would be just to use a different nxz for each mesh (and set an uniform id to identify them in picking). If the meshes are different connected components, it would be very similar to rendering a single nxz, if instead the meshes are just part of an object (like an arm of a body), cracks would appear in the middle.

The uv building process split and rearranges the uv components expecting it to be present in the ply/obj and changing it values rearranging the atlas, I don't think it's possible, to exploit it.

Maybe it's possible to use the alpha component of the colors, which should be preserved during the simplification and compression, with the obvious problem of simplification between vertices with the different ids, but it could be a start.

Another solution still could be to use textures: you assign each mesh a different material, a parametrization (doesn't matter wich) and a set of small (2x2?) textures, each with a different color. The simplification will preserve the boundary (never simplify across), you will have to access the sampler to get back the id.

Adding id attributes is planned, but needs rewriting a lot of things....

[ua4192]

Hi.

Many thanks for the info.

I am already trying texture aproach, I will keep you informed.

Best regards