Smart Fill: aka. LazyBrush, automatic paint

[b4zz4]

Painting layers or drawings remains complicated in blender. I was trying to paint using skimage simulating Lazy Brush

Although I wouldn't know how to do this in blender

https://dcgi.fel.cvut.cz/home/sykorad/lazybrush.html https://docs.krita.org/en/reference_manual/tools/colorize_mask.html

from scipy import ndimage as ndi
import matplotlib.pyplot as plt

import numpy as np

from skimage.morphology import disk #, binary_dilation, dilation, skeletonize
from skimage.segmentation import watershed
from skimage import data
from skimage.filters import rank
from skimage.util import img_as_ubyte
from skimage import io

image = img_as_ubyte(io.imread("in.jpg"))

# denoise image
markers = rank.gradient(image, disk(4)) < 20
markers = ndi.label(markers)[0]
gradient = rank.gradient(image, disk(2))
labels = watershed(gradient, markers)

# display results
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(8, 8), sharex=True, sharey=True)

ax = axes.ravel()

ax[0].imshow(image, cmap=plt.cm.gray)
ax[0].set_title("Original")

ax[1].imshow(gradient, cmap=plt.cm.nipy_spectral)
ax[1].set_title("Local Gradient")

ax[2].imshow(markers, cmap=plt.cm.nipy_spectral)
ax[2].set_title("Markers")

#ax[3].imshow(image, cmap=plt.cm.gray)
ax[3].imshow(labels, cmap=plt.cm.nipy_spectral, alpha=.5)
ax[3].set_title("Segmented")

for a in ax:
    a.axis('off')

fig.tight_layout()
plt.show()

base in:

https://user-images.githubusercontent.com/643833/221452021-a514b026-23ac-45a9-9d23-d97fa8ff3741.mp4

[chsh2]

I think it is a common difficulty to apply these pixel-based algorithms because we cannot easily get pixel information through Python APIs.

Maybe one way is to render the image and then use skimage.measure.find_contours to convert the image back to paths. But I can expect a lot of problems during these conversions, and it will certainly be slow.

Or we can change the algorithm from pixel-based to polygon-based. I saw a new technique Delaunay Painting that aims at the same effect as LazyBrush, but uses triangles instead of pixels.

I think both directions are worth investigating. I prefer the second approach for this feature, but the first approach may be necessary somewhere else such as color image import, which I have not found an easy way to implement.

[b4zz4]

Now I need to know how to propagate the color and unite the triangles to form one form

[chsh2]

Now I need to know how to propagate the color and unite the triangles to form one form

I think the algorithm to propagate the color, or to segment the graph is the most crucial part here. I have some methods in mind but do not know if they work well visually, unless I try them. I also need to finish reading the paper.

To unite the triangles, the Bmesh module has an attribute is_boundary that can help us find the contour of the shape. We can connect the contour vertices together and ignore others.

[chsh2]

I am working on this feature now and have made some progresses.

The method to propagate the colors is maximum flow. The concept is straightforward, but it needs multiple stages of data structure conversion. Therefore, I may need some more time to get it finished.

[b4zz4]

i think it would be good to be able to paint with little filling in a "hints" layer and transfer the filling to the other as a smartfill. So do not create new fillings of vertices.

[chsh2]

i think it would be good to be able to paint with little filling in a "hints" layer and transfer the filling to the other as a smartfill. So do not create new fillings of vertices.

I am not sure if I understand this idea correctly. These are some of the current features of Smart Fill:

• Fill materials can be used as hints, but currently only vertex colors are considered. I need to add a new option to use fill colors instead.
• The algorithm can only create new shapes. It cannot transfer existing shapes. But there is an option "Clear Hints" to delete old strokes. The result will be similar to the transfer.

If this is not what you mean, do you have an example?

[chsh2]

I understand it after reading your commit. I think it is the same question discussed in #26.

I will merge the color picking commit now. As for the material one, it needs more changes in other places. I will handle it together with #26 soon.