EmbarkStudios / texture-synthesis

🎨 Example-based texture synthesis written in Rust 🦀
http://embark.rs
Apache License 2.0
1.77k stars 84 forks source link

Proper alpha channel support in matcher. #31

Closed vi closed 5 years ago

vi commented 5 years ago

Currently alpha channel is ignored in matcher algorithm. It is used just in final phase, leading to illusion of alpha channel support.


Some comments by @austinjones from #22:

@vi @Jake-Shadle I have a prototype of generic pixel support (RGB/RGBA/Luma/LumaA) implemented on austinjones/texture-synthesis/multipixel, but it needs some work. It's just a quick exploration.

I tried handling alpha this way: Premultiply alpha and only consider color channels. This makes sense when later compositing: bright red with very low alpha and bright green with very low alpha are actually similar colors when added to anything. I didn't add alpha to the cost computation though, because the correlation with the color channels will cause bias in the cost metric.

There were some side effects:

* Session requires a type parameter (the pixel type).  It has to be specified by the user whenever SessionBuilder::build() is called.

* There is a decent performance hit.  Need to figure out why...

* The guided synthesis test fails.

Let me know if you are interested in the code. I can try to fix it up.


@vi Yes, you are right! I was trying to solve a different problem (support for example transparency).

I think what is supported right now is cost computation based on RGB, but pixel copies are on the full RGBA. If you add a bunch of transparent bright red, the cost is computed on bright red. Take a look at k_neighs_to_color_pattern - note the multiplier on the first line is 3, not 4. Transparent bright red is considered bright red when running the matching algorithm.

vi commented 5 years ago

As for me, alpha channel can serve as: