Multi-camera texture projection

[Dekker3D]

Is your feature request related to a problem? Please describe. The current texture projection, even in depth+image mode, doesn't seem to be very good at continuing from a previous projection at a different angle. So it's great for assets that will only be seen from one angle, and kinda bad for assets that will be seen from many angles.

Describe the solution you'd like Let the user set up multiple cameras, list them in the texture projection panel, and generate a texture from all of them at the same time. The way I'd do this:

• Generate a "visibility map" from each camera, where the R and G channels indicate a pixel's position on the texture, and the B channel has a dot product so that pixels facing the camera get a stronger weight than those facing sideways.
• Generate a texture in small steps: for each camera, do the first 5% of steps, then blend these (noisy) images together using the visibility map's channels. Take new screenshots or renders from this combined texture, for each camera.
• Continue another 5% or 10% of steps from each camera, using the now-combined texture. Blend the new result together again, rinse and repeat.
• The first steps seem far more important in determining where features will be, and you don't want to leave latent space too often (because that's slow), so this loop should use fewer steps at first, and more steps later in the process.
• You should now have an image that matches your prompt, as well as SD can make it, from every camera angle.

[dddjef]

I thought about the same thing, but without camera. Have you heard about triplanar mapping? It could be something a bit like this, but cubic. We compute a projected map for each 6 axis, mixed with face normal. It could even be baked at the end on standard non overlapping uv. I love your suggestion with "stepped" denoise to help to converge the results.

[abrasivetroop]

If this gets implemented it would be a huge game changer in indie game dev sector. This would cut costs and save a lot of time for indie devs creating assets. I was considering updating to new version of dream textures because I saw the new "texture projection" feature but then I realized it is best suited for assets that are gonna be seen from only one angle :/

I don't really have any experience with AI image generation but as an artist and a game developer I would ABSOLUTELY LOVE if this feautre comes true!

[p4vv37]

I recently tested a similar idea: The rendering of objects with SD from multiple views and blending results for texture generation. Some results are not terrible :) You can check out my, very crude, implementation. The code is extremely simple and not optimized, but possibly I could try writing something similar on a branch here what do you think?

[Dekker3D]

That would definitely help! So far we haven't had much success, and the current projection technique has that odd offset to it.

[abrasivetroop]

I recently tested a similar idea: The rendering of objects with SD from multiple views and blending results for texture generation. Some results are not terrible :) You can check out my, very crude, implementation. The code is extremely simple and not optimized, but possibly I could try writing something similar on a branch here what do you think?

That's really impressive! Dzięki :p

[carson-katri]

@p4vv37 that would be great! The main projection code is here: https://github.com/carson-katri/dream-textures/blob/main/operators/project.py And the depth pipeline is here: https://github.com/carson-katri/dream-textures/blob/main/generator_process/actions/depth_to_image.py Anything that uses libraries other than those built-in to Blender will need to be run from a function on the generator, like def depth_to_image.

[p4vv37]

Thanks! The code is nice and it'll take me a while to make my stuff compatible with it :) But I'll try doing that

[Dekker3D]

In case you weren't aware yet, carson's attempt is at https://github.com/carson-katri/dream-textures/pull/494 and the corresponding branch. It results in mostly single coloured textures with some blurry bright or dark dots in places, and I haven't been able to figure out the problem either. The UI element for choosing camera angles is definitely nice though, probably better than just having it rotate evenly around the model.

[Dekker3D]

I had some further thoughts on this. Concave models (with potentially overlapping parts) could be automatically masked using some sort of "camera visibility map". My best guess at how to do this, is to simply use the depth map and check whether a pixel is at the distance we expect it (indicating it's the surface facing the camera), within some threshold.

I personally still feel like the best way to do multi-angle texture projection is by directly projecting to the model's UV mapped texture, rather than projecting an image from each angle and blending them together. This does, however, require some way of looking at the original texture coordinates and finding the matching camera-space coordinates for it, and back. The "back" is easy: just make a render with the UV coordinates making up the red and green brightness for each pixel. The former, though? Not sure.

[p4vv37]

I saw that #494 exists, but did not yet have time to check out the code changes. I just started to read the code and analyze the structure. Camera facing would probably be easier to do, by checking the normal direction, if it's needed.

In the meanwhile, I created a UI for my code (when I understood how much I'd have to change it to make it fit here :) ) https://github.com/p4vv37/SDTG4B The results are not perfect, but definitely better than just dots. I noticed, that depth2img for some reason even with strength 1.0 is very dependent on the provided image. This is why a totally black input image produced bad results, maybe that's the problem?

The way you described is more or less how my implementation work. In the middle of the video I presented is as a graph: https://twitter.com/Uconeni/status/1617295370228228096?s=20&t=vH7U5Tj_gQJ-7asHF0B5IQ

[Dekker3D]

Using the normal direction to check camera facing will work fine on convex meshes, but when you have overlapping stuff (like two legs, from the side) it would draw on both legs, right? This would result in garbage output on the overlapped surface, which is undesirable, especially if trying that "blend between steps" approach I suggested for consistency.

[p4vv37]

Ok, I understand why we could not communicate, I was still thinking in terms of the render, I see now that you're doing it the right way, in code, without rendering :) For my demo project, it was a given. Right, some form of occlusion culling will have to be implemented. I'm not yet sure what will work best, but maybe creating a buffer with depth will be necessary. But there might be another way: Blender does it for every tool when the "toggle X-Ray" button is disabled: tools work only on surfaces visible from one point. So maybe such a mechanism is available somewhere in the code and it would be enough to find it and hack it for our needs. This need to be checked.

By the way, using textures generated for previous view also helps, so it might be good to render such a picture as well... In fact we're getting closer to the idea of a simple rendering engine at this point...

Anyway, I need to start by studying the code.

[p4vv37]

Unfortunately, no. It does occlusion on GPU from data collected during viewport rendering. Still, something might be figured out, Blender supports some structures that might help: https://docs.blender.org/api/current/mathutils.bvhtree.html

[Dekker3D]

So an article just made me realize how to do the inverse UV lookup texture. https://echoesofsomewhere.com/2023/01/25/modeling-a-robot-with-some-added-difficulty/ talks about "morph maps", which are just UV maps that are modified to match a texture. Not directly related to this problem, BUT!...

We can use Geometry nodes to first store the coordinates of each vertex, then reshape the original mesh to look like its UV map (all flattened out), and colour each vertex based on its original coordinate, remapped to the camera space (which can be done in shader nodes, after feeding it the original coordinates as colour attributes or something).

When you have that, you can easily map pixels back and forth between camera space and texture space, which should allow all sorts of things to be tried. But setting this up automatically will be quite a pain, I think... Lots of data to juggle and then clean up, to avoid polluting the .blend file.

[carson-katri]

That sounds like how the auto bake works: https://github.com/carson-katri/dream-textures/pull/495 Except it just draws it all with Blender’s GPU module from Python, no geometry nodes or anything.

[Dekker3D]

I hadn't realized that. That should help a lot!

[Dekker3D]

Oh, one way to get consistency across sides in your model is to just show more angles of the model for each shot. So for the treasure chest example, you'd put some linked duplicates side by side, each one rotated at a different angle. That way, it probably should give them similar textures, right?

[p4vv37]

I'll try to find some time to look at this a bit deeper this week. Possibly making duplicates would work, bt possibly just creating e.g. 4 views would work fast enough

With this said, this paper looks way more promessing than my solution: https://texturepaper.github.io/TEXTurePaper/

[p4vv37]

I checked the paper and the idea is very similar but more complex. Instead of depth-based masking normal-angle-based masking is used, but it is used in a more complex way. This mask is used to modify the denoising process instead of just blending the result. There's also an interesting way of handling "refine" (Where normal angle is high, but not too high) areas by checkboard-based masking. They also use a combination of inpainting and depth2img in a clever way, that's a nice idea. Anyway, those changes are SD-side, not Blender-side, so after having a correctly Blender-side part it would be nice to check them out and maybe apply some of them. The Blender side of the process would be the same.

[Dekker3D]

Yeah, I read through it too. If you can, please come to the Discord so we can discuss things more easily? https://discord.gg/EmDJ8CaWZ7

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale.

[krazyjakee]

I thought this was part of the next milestone?

[Dekker3D]

I thought this was part of the next milestone?

A bot has been closing many issues as stale. The devs aren't specifically involved in closing this issue.

[enzyme69]

github-actions bot needs to be turned off, I think lots of "issue" and "wishlist" closed without notice.