search

fmu2 / 3d_photo_stylization

3D Photo Stylization
107 stars 15 forks source link

I try to use 3DP to obtain LDI data, and operate your model to stylize my onw image. Getting bad result #4

Open shoutOutYangJie opened 2 years ago

shoutOutYangJie commented 2 years ago

https://user-images.githubusercontent.com/46734514/179400328-1f038e1e-2114-437a-b20e-104c333df202.mp4

First, the ratio of aspect is not correct. Second, a black area exists in the boundary of image. Third, the quality of stylization is not perfect as in your demo.

I want to know how to solve it, thank you.

fmu2 commented 2 years ago

It looks like either the LDI parameters or the test flags were not set correctly. I would suggest the following sanity checks one by one:

You may want to experiment more advanced depth estimation models if the aforementioned steps do not fully solve the problem. As you may have noticed, the output quality strongly depends on depth accuracy. We recommend using DPT for the best depth estimation result.

Finally, some scenes are harder to stylize than others, and not all styles give equally good results, so try a handful of styles. My impression is that your scene is a hard one because it contains lots of thin structures (e.g., trees and fence). Even the best depth estimation models fail on this type of images.

shoutOutYangJie commented 2 years ago

It looks like either the LDI parameters or the test flags were not set correctly. I would suggest the following sanity checks one by one:

  • run ldi_render.py to see if you can get the right video without stylization. This makes sure that your LDI has the right format. You may also want to tune the x, y and z bounds if you observe any black margins. The default values generally work well but depend on the depth estimation model and the aspect ratio of your input image.
  • try different scales for your LDI using the -pc flag. As a rule of thumb, set it to round(max(h, w) / 500) where h and w are the height and width of your input image. Run ldi_model.py without a style image with your chosen -pc value. Make sure the video looks close to what you obtain from the first step.

You may want to experiment more advanced depth estimation models if the aforementioned steps do not fully solve the problem. As you may have noticed, the output quality strongly depends on depth accuracy. We recommend using DPT for the best depth estimation result.

Finally, some scenes are harder to stylize than others, and not all styles give equally good results, so try a handful of styles. My impression is that your scene is a hard one because it contains lots of thin structures (e.g., trees and fence). Even the best depth estimation models fail on this type of images.

Can you share the preprocessing code. Your work is very great. If you can share your preprocessing code, the citation may be more and more increasing.

fmu2 commented 2 years ago

Thanks for the suggestion. The preprocessing code relies on multiple third-party tools. We will release it after some cleanup. Please check back in a few weeks.

shoutOutYangJie commented 2 years ago

I look forward to it. Thank you

------------------ Original ------------------ From: Fangzhou Mu @.> Date: Wed,Jul 20,2022 0:24 AM To: fmu2/3d_photo_stylization @.> Cc: shoutOutYangJie @.>, Author @.> Subject: Re: [fmu2/3d_photo_stylization] I try to use 3DP to obtain LDI data, and operate your model to stylize my onw image. Getting bad result (Issue #4)

Thanks for the suggestion. The preprocessing code relies on multiple third-party tools. We will release it after some cleanup. Please check back in a few weeks.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>