feature request: directly generate OpenGL texture output

[diablodale]

A new feature request. To output opengl textures rather than jit matrices. Good for patches that only process things in jit.gl.* objects or in shaders. This will remove one process from the chain of efforts and should cause a slight improvement in overall patch performance (though may not be measurable).

The simplest implementation would be a single attribute that designated that all matrices outlets will instead output textures. A drawback of this is that it doesn't allow a mix of matrices and textures (e.g. depth and color as texture + player and IR as matrix). With this simple approach a matrix can be created by chaining a jit.matrix object for only those that are needed.

Output types per data feed should be respected. For example, if @irtype=float32, then the IR texture should be a float32. If @irtype=long, then the texture should be a single plane 32-bit long texture.

It is not planned to internally combine multiple textures into one. Shaders and other tools can easily accept multiple textures and can do any combining necessary in the vertex, geometry, or fragment portions.

@singularmoments do you have any comments on this texture feature idea?

[singularmoments]

I've been testing the v1.1. and particularly the align=2

I still think it's not the optimal solution to map the depth to color and upscale it to 1920x1080. It generates for (depth, user, cloud) around 9 same values (3 on x and 3 on y axe) for nothing and it will incurs in an x9 processing for the gpu. It also adds artifacts and bigger border around users cos the cloud is mapped from an offset camera_color center.

I think the optimal solution is the one used by the app kinect studio. it textures the color (1920x1080) on the cloud (512x424), simply. I suppose it gives each point of the cloud a texture coordinate (xy:0. > 1.) in the color map. It's the only solution to stay with the fidelity of the point cloud for futher processing and to texture FHD color.

for this we absolutely need access to this tex_coords.

The solutions for dp.kinect2 ? 1 _ add an outlet for texture_coordinates (x y) with a jit.matrix 2 float32 512 424 (val from 0. > 1.) I suppose that for align = 1 , you use MapDepthFrameToColorSpace it returns you a map with coordinates in the color map. Perhaps you can write this TC map directly with those coordinates / (1920,1080) and jit.gl.mesh will take those values perfectly in its second inlet.

The mode will then stay align = 0 but with TC output.

Given that this solution (adding a TexCoord outlet) will break compatibility with older patches, perhaps you could output it through the IR map outlet and add an attr for chosing IR or TC output for this outlet.

Thanks for the hard work Patrice

[singularmoments]

Comments on map bang order:

I noted that when you bang the object, it outputs in order : data cloud user color depth (last)

for gl processing, its better to have an order: data color user depth (TC) _cloud (last)

I made a little patch that reorder the maps on each bang but it will be great to have this order with the cloud outputed last.

[singularmoments]

Comments on textures output:

I see that you made a RGBA mode for the color map. You are certainly aware of the fact that passing from matrix to texture in a patch swap argb to rgba. Its the same for xyz cloud. i made a little jit.gl.gen patch to reorder the planes. (swiz.wxyz) Would you take that in consideration for outputting all maps to textures directly. I think also that for futher processing, user should be outputted in float32 texture ( 0., 1., 2. ...)

Perhaps I could send you some patches to show you my worflow all in gl world: xyz transform and temporal softening normal generation (taking depth break into account) user selection from the cloud xyz user normal _ color encoding and decoding in one texture (4 char 2560 424) to pass it with spout geometry shaders for the cloud

[diablodale]

Thank you for your ideas and questions.

• The Max standard for all objects is for outlets to have right->left ordering. If you want different order, it is relatively easy to use objects like (trigger), (buddy), or (swap) to change it for your needs. If you have a strong need for dp.kinect2 to do this internally, please provide a patch or specific use case so I can better evaluate this.
• beta v1.1 supports ARGB, RGB, and RGBA output. Please look at the new values for @colormap in the Max inspector
• beta v1.1 supports char and float32 output. Please look at the new values for @colortype in the Max inspector.

Regarding the alignment...

• There is no upscaling or downscaling. There is only mapping. For example, if you use @align=2, then as I am creating a 1920x1080 depth matrix, there will be several 1920x1080 depth pixels that map to the same native 512x424 depth pixel that is aligned to the 1920x1080 color pixel. The mapping isn't linear or a simple up/downscale. You would not generate good results if you tried to overlay the raw depthmap directly over the colormap...or the opposite. Therefore, the align mapping process is needed to generate correct pixel-to-pixel alignment.
• App kinect studio and dp.kinect2 do the same thing. If you want "it textures the color (1920x1080) on the cloud (512x424)", this is done with @align=1.
• Artifacts, boarders, etc. will be true in all situations of the raw data. This is expected data behavior with a two camera depth and rgb sensor that are offset from each other -and- can't see through solid objects. It is the responsibility of the patch to digest this data knowing these limitations in physics and to do their own filtering, masking, etc.
• I haven't understood from you a need that isn't already met with @align=1 or @align=2. Using either of these options will get you pixel-to-pixel alignment of both depth and color. Can you provide a simple patch or pseudocode or screenshots that can help me better understand your need?

[singularmoments]

Well, the point is that there's only 2 optimal mapping scenario :

1 _ cloud 512x424 (undeformed) color 1920x1080 undeformed, mapped with texture coordinates on the cloud

2 _ cloud 512x424 (undeformed) color 1920x1080 depth_mapped,

that's what is doing kinect studio, if you look closely, the monitor 3d view, the cloud is undeformed (512x424 and not mapped from color or 1920x1080), and the color is 1920x1080 mapped (and not 512x424) you can change the view in the button parameters.

at the moment, your 3 solutions output:

align= 0 : cloud OK _ color not ok : FHD possible but no depth_mapped

align = 1 : cloud OK (512x424) _ color not ok: depth_mapped but not hires (512x424). if colorres could be 1920x1080 and depth_mapped it will be perfect without texcoord needs.

align=2 : cloud not ok (mapped to color and upscaled) , color 1920 but missing tex coord

Once again, if you look closely at monitor 3d view, you can not obtain the same quality, with whatever mode of dp.kinect2

[singularmoments]

The point is that pixel to pixel alignment between cloudmap and colormap is not the optimal solution. at least its not the one used by monitor 3d view.

when you map a video texture onto a gridshape plane that is dim 10x10 you do not provide a video that is 10x10 pixel you provide a bigger video which interpolate between vertices

in our case, you will map a 1920x1080 color map onto a 512x424 cloud but contrary to what you stated, your align=1 does not allow a 1920x1080 color map (depth_mapped)

[singularmoments]

where can i send you patches?

[singularmoments]

Comments on map bang order 2: It just that when you process the maps, its more logical to receive in an object the cloud in left inlet and the user or color in the right inlet. a bit like jit.gl.mesh that will receive the cloud last in the first inlet. yes right to left order I made a reorderring trigger patch so i already live with it :)

[diablodale]

You can attach patches directly here. Instruction are below the edit box, "Attach files by..." I just used Kinect Studio v2 and here is a screenshot. Is this what you are referring to? If so, then this same 512x424 output can be generated with v1.0.1 and HD output with v1.1 beta.

[singularmoments]

yes perhaps a picture is better; look at the text on the book. the quality difference is huge

as you can see, align=1 the cloud not modified is the same as in kinect studio. the only difference is that the color is 1920x1080 depth_mapped.(instead of 512x424)

align2 kind of kill the cloud , being color_mapped.

If you want to see it by yourself, a small patch.

kin2_cloud_test.zip

[diablodale]

Attached is a simplified patch that uses v1.1 beta features. kin2_cloud_test2.zip You can also get rid of the (buddy) if you can live with an occasional 1/30 second sync mismatch. Does this give you the visuals you want?

You write at the beginning of this issue that the pointcloud with @align=2 generates a cluster of same xyz values. This would suggest that the Microsoft APIs doesn't interpolate or prevent such duplicated x,y,z values.

This duplicated data could create hidden triangles. Triangles with an area of 0 -or- triangles that are in the exact same place as other triangles. These hidden triangles would have their own HD pixel colors. However, since they are hidden, we never see those colors rendered in OpenGL. Instead, only the colors on the visible triangles would be rendered. So the data is there in the colormap and sent to jit.gl,mesh but the affect due to the duplicate xyz values is that we only see some of the colors.

If this is the case, I have a few ideas. They are in order least->most work done by dp.kinect2 and inversely most->least patching complexity/advanced knowledge

1. When dp.kinect2 detects duplicate data, it replaces the duplicates with linear interpolated data. There is some complexity here dealing with scan rows, YUV2, etc. All users of dp.kinect2 would transparently benefit from this approach. No texturing, shaders, or other advanced patch knowledge is needed.
2. Create texture coordinate maps. For example, align=1 would create a textcoord matrix with a size=512x424 with 2 planes. Each cell of this matrix would specify a UV value [0.0-1.0] which represents a location in the colormap. There is some complexity here to handle the unmappable/unalignable pixels and the edges. This would likely make more color detail appear if implemented correctly in the patch. This requires more advanced patching knowledge but not to the even higher level of writing a geometry shader.
3. Defer this work to a geometry shader. A custom shader written by a patch author could ask for a cluster of triangles rather than just a single triangle. The geometry shader could look for duplicated data. If it is found, then the geometry shader creates subtriangles which are interpolations of the duplicated data. There is some more complexity in this approach. And it would be difficult for most Max patch authors.

[singularmoments]

I start to understand where we differed. You plan to input the color map as a matrix in jit.gl.mesh (color array for vertex) when i think it should be used as a texture like in the first patch i send you. I think kinect studio use the texture solution. its also much more efficient in processing: colormap > texture

Solution 1: For me duplicating or interpolating or mapping the original cloud is an error. it does not have to be. it must stay 512x424. re-interpolating to 1920 a color map that you first mapped to 512 is also a no go.

Solution 3 : not necessary

your solution 2: outputting TC is the correct one as i suggested in my first post, in this case the colormap must also be outputted but non mapped in 1920x1080. so its align = 0 but with an supplementary tex_coord output

it will be the best for a simple use of jit.gl.mesh an original cloud unmodified > inlet 1 an original color map in 1920 unmodified > to texture for the mesh object _an texcoord matrix that automaticaly map this texture onto the mesh > inlet 2

I could also use this TC to mix the player map and the color map even if they got different resolution. In this solution 2 of TexCoords, the align= 1 and align=2 mode becomes unnecessary.

solution 2B: allow align=1 to output a 1920x1080 color map (depth_mapped) . (at the moment its only 512x424 depth_mapped) in the max patch, giving simple normalize texcoord to jit.gl.mesh like in my patch, the bigger color TEXTURE will map onto the cloud. That's why i suggested you in my 1st post: supposing that you use the: MapDepthFrameToColorSpace, and that it gives you coordinates in the color map instead of using those coordinates for generating a 512x424 color map, generate a 1920x1080 map ( or 1024x848 or 1512x1272) interpolating on the colormap is not a problem given that it's continuous, interpolating on the cloud is problematic.

thanks for the attention. Patrice

[singularmoments]

Hi Dale, I'd just like to know if you are planing to implement in the near future the tex_coord output we have been talking about. (or the color 1920x1080 depth_mapped)

As your perhaps understood, i'm finishing a tool for encoding in 8bits the kinect geometry (cloud+user+normals+color). This way i can send it through spout (to another app) or HDMI 4:4:4 (to another computer).

Thanks, and no pressure ;) Patrice

[diablodale]

The v1.1 beta is feature complete. At this stage of development, I am focusing on testing, final optimizations, and documentation.

This texture output idea is a good idea and I would like to implement it in a future release of dp.kinect2. That future release will probably not be ready until late Spring or early Summer.

[singularmoments]

Oki, I'll try to map the FHD trough perspective corrections. keep the good work Patrice

ps: Ich habe in die neunziger im Ost Berlin gewohnt; Echt verkerht Ich hoffe das berlin ist so gut wie in dieser zeit. Vielleicht wir sehnt dort, wann ich eine paar freunde zu besuchen ...

[diablodale]

Gerne. :-) I did some preliminary research into direct textures. To get a significant performance improvement over what is already available today (output, patchcord, jit.gl.texture object), I have to use native low level OpenGL APIs and interface them with the OpenGL support in jitter. This is a large work item and will require significant testing. I think this is a feature for a summer release.