3d Texture Synthesis with Customized Input Texture

[Talk2Levi]

I really love the feature of the 3d Texture Synthesis, and I've been trying during the past few days to pass my own customized texture as the style to synthesis on the 3d model into the neural net instead of using the visualization of a neuron channel.

What I understand so far is that these two lines of code:

• obj = objectives.channel('mixed4b_pool_reduce_pre_relu', 17)(T)
• tf.losses.add_loss(-obj)

is where the texture got passed in, and really what this does is get the mean of the tensor (placeholder) which equals the code below:

• obj=tf.reduce_mean(sess.graph.get_tensor_by_name("import/mixed4b_pool_reduce_pre_relu:0")[..., 17])

one thing I'm curious about is the obj is the tensor with shape = () and you could still render that specific obj with

• lucid_render.render_vis(model, obj, verbose=False), the rendered result is an image with shape = (1, 1, 128, 128, 3).

I have my own image with that size already, instead of pulling the visualization of the layer out, what I really want to do is pass the image into the neural net and synthesis with that image on the 3d model.

I know there must be a way to do it, and I probably just being dumb to not figure it out😂, so please if anyone could help me with this, I will be really appreciate it since I've been working on it for almost a week.

Thank you guys so much, and have an awesome day!!!

[ludwigschubert]

Are you maybe just looking for the Style Transfer Notebook instead?

[Talk2Levi]

Thanks for replying to me!!!

For that, yes and no. I've tweaked with Style Transfer Notebook longer than the 3d Texture Synthesis until I realized the 3d Texture Synthesis could potentially solve my problem better easier (not sure about this atm though 😂).

For the Style Transfer, the model really learns the pattern & the color & all the other features of the original texture, for instance on the Stanford Bunny, after the style transfer, the new bunny still has a darker region around the eyes and the whole body tends to be red-ish even with new style on that in order to match the original texture as close as possible which really is amazing!

But the only thing I need is to keep my customized texture as what it is and ignore the feature on the original texture of the 3d object, essentially just cover up (or synthesis on) the 3d object with the customized texture and most importantly to achieve seamlessly which is just like what you guys did in 3d Texture Synthesis but it's just that one thing (i.e. the original idea of this issue, pass my own customized texture into the NN) blocked me for a little while now.

I'm really sorry that I didn't clarify my problem well enough, I hope this answer would help, and please let me if you need me to clarify any other things, and I will be so grateful if your team could help me or direct me what to do.

I really appreciate it!!

[ludwigschubert]

Sorry, I either don't understand, or you're missing an actual objective to optimize. "essentially just cover up (or synthesis on) the 3d object with the customized texture" -> you'd need to put this statement into an objective function that can be optimized.

---

Please use the Stack Overflow tags tensorflow & lucid to ask for help. Stack Overflow provides a better collaborative forum: lots of related questions have been asked there, and some answers may be relevant to you.

When asking for help with a suspected bug, please include a link to demonstrate the issue, preferably as a Colab notebook. It is often impossible to debug from code snippets alone, let alone from error messages or stack traces. Isolate the issue and reduce your code as much as possible before asking for help. Please try to reproduce the bug on one of the models included in lucid.modelzoo. The less code you post, the easier it is for someone to debug, and the more likely you are to get a helpful response.

If you have a question about Lucid’s behavior and want to discuss it with other users, also consider the #proj-lucid channel on the Distill Community Slack Workspace which you may join here.

Lucid is alpha-status research code, and we appreciate your understanding that this can sometimes mean we will not have the resources to resolve your specific issue—even if it may be an entirely reasonable request in a more mature codebase. Thank you! 🤗