slow capture due to rage_bonemtx

[el3ment]

All of the computation in startDraw for rage_bonemtx slows down the render loop quite a bit. In particular

https://github.com/philkr/gamehook_gtav/blob/master/gta5.cpp#L301

I haven't found any code in the repo that uses this data - is it necessary?

[philkr]

It is used for the optical flow computation of rigged objects (e.g. people, animals). Unfortunately, optical flow is currently the main bottleneck in terms of performance. It takes about 10-20ms/frame on my machine, depending on the complexity of the scene. If you have any ideas on how to speed it up please let me know.

[el3ment]

I'll see what I can do - is there a reason you are copying those matricies in the first place? it seems like all of the data you need (namely just the vertex transformation matrix and projection matricies) should already in the shader.

[philkr]

For animated objects, I compute flow (location a certain vertex had in the previous frame) using the bone matrix of the previous frame, plus the previous world and projection matrices.

On Thu, Jul 12, 2018 at 12:26 PM Robert Pottorff notifications@github.com wrote:

I'll see what I can do - is there a reason you are copying those matricies in the first place? it seems like all of the data you need (namely just the vertex transformation matrix and projection matricies) should already in the shader.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/philkr/gamehook_gtav/issues/2#issuecomment-404588115, or mute the thread https://github.com/notifications/unsubscribe-auth/AAd3rmd5ghzxO-bNPhlJu1nqsSVaoAwwks5uF4bTgaJpZM4VL7aJ .

[el3ment]

I guess I just don't see where that computation is happening. The closest thing I can find is https://github.com/philkr/gamehook_gtav/blob/master/vs_static.hlsl#L6

[philkr]

The actual computation is a bit opaque at the moment. Most of it is in injectShader, which produces a shader to append to the original game shader. https://github.com/philkr/gamehook_gtav/blob/master/gta5.cpp#L111 duplicates a vertex shader of the game and renames its inputs and outputs, before appending it to the original shader. This will duplicate the vertex shader code. Once it renderes using the current parameters (bone mtx, ...), once with the previous ones. The previous position is then stored in PREV_POSITION, which is used in https://github.com/philkr/gamehook_gtav/blob/master/ps_output.hlsl .

[el3ment]

Ah. Maybe I am missing something - but doesn't the vertex shader know PREV_POSITION when it starts attempting to render using the current parameters? why not just write that position out for the pixel shader, as opposed to recalculating it?

[philkr]

I'm not sure I see how a vertex shader knows the position of a vertex in the previous frame, when rendering the current frame? The shader is stateless.

[el3ment]

the "previous" position of a vertex is the "POSITION" at the start of the shader, isn't it?

current_position = transformation * position

the pixel shader only sees "current_position" - but the vertex shader knows both.

[philkr]

yes, for any static objects you can compute the prior position using the previous transformation matrices. However, animated objects change vertex positions between frames (controlled by the bone matrix).

[el3ment]

Sorry - I understand now. I've gotten started on trying to speed up these calls, but as I've been trying to debug the flow vectors I'm realizing that they are not quite right yet such as (for example) parked cars not being present in the flow-maps.

It looks like pos and prev_pos in ps_output.hlsl are quite different from one another and I'm not sure what coordinate frame either of them are supposed to be in. I'm suprised you don't compute flow by simply subtracting the two prev_pos - pos.

UPDATE: I am fairly certain something is happening during the merge of the replicated shaders - for example a register is being update in place, and then reused during the replicated part of the shader (only this time with a non-zero value). This seems likely since prev_pos and pos should be identical when prev_rage_matrices == rage_matrices, but I am seeing very different values.

[philkr]

This is odd. Can you tell me what version of GTA you're using, what which shader and what inputs cause this. I'll look into it.

[el3ment]

so this is/was happening because I am capturing all of the items on screen, instead of the limited few you were capturing before. the primary issue had to do with how the object tracker identified "similar" objects -- I needed to add "texture" as a differentiation factor. there were a few other smaller things too, but this was the biggest one.

[el3ment]

@philkr what is the disparity correction term for? to account for changes in the projection matrix that might affect depth values? did you notice that happening with GTA5?

Also - were you ever able to get GTA5 to run in DX11 mode?

[philkr]

I fixed the bug that prevented dx11 in gta in https://github.com/philkr/gamehook/commit/08042b24e414801ec4d43f75e34ce56a9c97e423.

As for the disparity correction: I wanted to get the original disparity (depth) before the projection matrix. The projection matrix is mostly fixed, but does slightly change at high speeds (the fov changes too there). However, that piece of code was buggy (and likely numerically unstable), hence the two constants.

I also just saw today that flow (and likely the parts of depth) are completely broken in v1.0.1365.1 , see #5 . I'll fix this once I have some more time. Feel free to close this, and we'll continue in #5.