brushes

[mastef]

Have you thought about using custom brushes? So instead of basic shapes, the algo could also try to apply certain custom shaped brushes to make it more paint like

Bonus points would then be to emulate brush strokes

Just an idea, not sure yet about format of brushes

[mastef]

I had this working somewhere actually, need to double-check.

[Tw1ddle]

I was wondering if this might be done by adding some way to add custom shape types, or perhaps an overridable polygon shape type. Do you have some code you can show me?

[cancerberoSgx]

I wonder if this needs to be done on each step or as part of the algorithm or if it can be done offine by a post processing tool that recreates the image.

In the later case a cheap way could be post processing tool to add svg image patterns to all shapres in the svg like https://stackoverflow.com/questions/11496734/add-a-background-image-png-to-a-svg-circle-shape..then use a svg rasterization tool to generate bitmap

if it needs to be on each step what about representing the brush with a second bitmap and use it to fill shapes in the rasterize() instead of just solid color? perhaps chossing random offsets for render it so the finav result is not "homogenous" (using the same color and alpha)

my two cents

[mastef]

I added a brush by specifying it in the getSvgShapeData :

geometrize/shape/Brush.hx

package geometrize.shape;

import geometrize.exporter.SvgExporter;
import geometrize.rasterizer.Rasterizer;
import geometrize.rasterizer.Scanline;
import geometrize.Util.Region;

/**
 * A rotated brush
 * @author Markus
 */
class Brush extends RotatedEllipse implements Shape {

    override public function clone():Shape {
        var brush = new Brush(xBound, yBound);
        brush.x = x;
        brush.y = y;
        brush.rx = rx;
        brush.ry = ry;
        brush.angle = angle;
        return brush;
    }

    override public function getType():ShapeType {
        return ShapeType.BRUSH;
    }

    override public function getSvgShapeData():String {
        var s:String = "<g transform=\"translate(" + x + " " + y + ") rotate(" + angle + ") scale(" + rx + " " + ry + ")\">";
        s += "<path d=\"M0.36-0.27c0.015,0.008,0.046,0.016,0.077,0.023c-0.046,0-0.062,0.002-0.107,0.002
    C0.344-0.23,0.36-0.22,0.375-0.208c0,0.002,0.016,0.006,0,0.01c-0.046,0.03,0,0.059,0.016,0.086C0.406-0.08,0.437-0.046,0.467-0.016
    c0,0.01,0.039,0.02,0.054,0.029c0,0.002,0,0.006,0,0.011c0,0.021,0.016,0.042,0.016,0.063c0,0.01,0,0.022,0,0.034
    c0.016,0.016,0.031,0.034,0.046,0.052c0,0.004,0.015,0.01,0,0.014c-0.016,0.013,0,0.026,0.046,0.036
    c0,0.002,0.016,0.005,0.016,0.009C0.66,0.264,0.737,0.296,0.737,0.33c0,0.025,0.015,0.054,0.015,0.08c0,0.01-0.046,0.02,0.016,0.032
    c0,0.002-0.031,0.008-0.046,0.012c0.031,0.014,0.077,0.028,0.115,0.04L0.813,0.496C0.783,0.494,0.752,0.49,0.706,0.486
    c0,0.02,0,0.039,0,0.052c0.015,0,0.015,0.002,0.015,0.002c0-0.002,0.016-0.004,0.031-0.008c0.1,0.01,0.115,0.021,0,0.032
    c0.031,0.008,0.131,0.012,0.1,0.025c-0.054,0-0.1,0-0.146,0.002C0.66,0.592,0.66,0.6,0.69,0.602
    c0.077,0.004,0.077,0.021,0.192,0.022c-0.031,0.002-0.069,0.004-0.1,0.006c0,0.01,0.131,0.012,0.1,0.026
    c-0.046-0.004-0.1-0.008-0.146-0.012C0.66,0.651,0.645,0.661,0.66,0.67c0.046,0.02,0.077,0.04,0.123,0.059
    c0.031,0.016,0.054,0.029,0.1,0.048C0.929,0.77,0.944,0.766,0.96,0.764c0.015,0.024,0.077,0.047,0.015,0.07c0,0.004,0,0.01,0,0.014
    c0,0.013,0,0.026,0,0.04C0.914,0.895,0.813,0.895,0.798,0.909c0.038-0.002,0.054-0.004,0.1-0.008c0,0.006,0,0.008,0,0.01
    C0.852,0.932,0.783,0.936,0.614,0.924C0.598,0.93,0.552,0.936,0.552,0.945c-0.016,0.008,0,0.018-0.016,0.025
    c0,0.011-0.016,0.019-0.031,0.03c-0.054-0.016-0.039-0.032-0.1-0.044c-0.062-0.014-0.015-0.03-0.077-0.044
    c-0.046-0.002-0.092-0.006-0.154-0.01c-0.023,0.016-0.13,0.02-0.208,0.031c-0.046-0.008-0.092-0.014-0.123-0.02
    c-0.039-0.008-0.069-0.014-0.177-0.006c0-0.01-0.016-0.019,0-0.026c0.015-0.014,0.031-0.026,0-0.04
    C-0.363,0.819-0.394,0.794-0.425,0.77c0-0.006,0-0.012,0-0.018c0-0.008-0.031-0.018-0.031-0.026c0-0.016-0.054-0.031-0.069-0.048
    c0-0.027-0.031-0.056-0.062-0.086C-0.602,0.546-0.648,0.5-0.602,0.454c0.016-0.021,0-0.04,0.062-0.058
    c0.016-0.007-0.015-0.013-0.015-0.021C-0.602,0.338-0.633,0.3-0.648,0.264C-0.664,0.236-0.679,0.21-0.694,0.182
    c0-0.01,0.016-0.021,0.016-0.031c0,0-0.031,0-0.046-0.002c-0.031-0.036-0.077-0.07-0.115-0.106c0-0.004,0-0.01,0.023-0.014
    c0.016-0.011,0.046-0.019,0.077-0.03c0.015,0.004,0.031,0.006,0.062,0.01c0-0.004,0.015-0.008,0.031-0.014c0-0.006,0-0.01,0-0.019
    c-0.016,0-0.062-0.002-0.108-0.004c0.077-0.023,0.031-0.048,0.016-0.075c-0.046,0.01-0.077,0.016-0.131,0.023
    c-0.031-0.034-0.031-0.068-0.046-0.1c0-0.019,0.016-0.032,0.077-0.046C-0.887-0.229-0.917-0.229-0.979-0.23
    c0.031-0.006,0.046-0.01,0.062-0.014C-0.902-0.24-0.902-0.236-0.84-0.238c-0.016-0.006-0.046-0.006-0.077-0.006
    c-0.015-0.014-0.107-0.027-0.015-0.044C-1.025-0.296-1.01-0.304-0.963-0.316c0.015-0.004,0.015-0.012,0.015-0.018
    c0-0.014-0.046-0.028-0.015-0.044c0.015-0.004-0.016-0.01-0.016-0.017c0-0.067,0.031-0.136,0.108-0.204
    c0.015-0.022,0.054-0.047,0.069-0.068c0-0.006,0-0.012,0-0.018C-0.817-0.69-0.84-0.695-0.787-0.701
    c0.016-0.002,0.016-0.006,0.016-0.008c-0.031-0.01,0-0.018,0.015-0.028c0.016-0.01,0.016-0.02,0.016-0.031
    c0-0.021,0.015-0.04,0.062-0.059c0.031-0.01,0.046-0.021,0.062-0.034c0.092-0.04,0.162-0.079,0.238-0.122
    c0-0.008,0.046-0.014,0.108-0.018c0,0.004,0,0.008,0,0.016c0.177-0.01,0.285,0.009,0.423,0.015c0.023,0,0.039,0.002,0.039,0.004
    c0.046,0.012,0.108,0.026,0.154,0.04c0.092,0.025,0.123,0.052,0.108,0.08c0,0.014,0,0.029,0,0.044c0,0.046,0,0.092,0.015,0.138
    c0.016,0.034,0.039,0.066,0.054,0.1c0,0.021,0,0.038,0.016,0.059c0,0.012,0.016,0.022,0.016,0.035c0,0.004,0,0.008,0,0.01
    c-0.069,0.012-0.069,0.012,0,0.024v0.002c-0.031,0.01-0.069,0.02-0.085,0.027c0,0.002,0,0.006,0,0.006
    c0.069,0.013,0,0.021-0.046,0.026C0.437-0.37,0.452-0.364,0.467-0.358l0,0c-0.092,0.013-0.077,0.026-0.031,0.04
    C0.344-0.306,0.329-0.29,0.375-0.276C0.391-0.274,0.375-0.271,0.36-0.27c0,0.004,0,0.008,0,0.014c0,0,0,0,0.015,0
    C0.375-0.26,0.375-0.264,0.36-0.27z M-0.633,0.088c-0.062,0.01-0.062,0.012,0.016,0.03C-0.571,0.108-0.602,0.098-0.633,0.088z
     M-0.648,0.026C-0.694,0.03-0.709,0.032-0.725,0.034c0.016,0.004,0.046,0.008,0.077,0.012h0.015
    C-0.633,0.04-0.648,0.034-0.648,0.026z M-0.709-0.258c0.031-0.004,0.046-0.006,0-0.012H-0.74C-0.756-0.264-0.756-0.26-0.709-0.258z
     M-0.664,0.084c0.031-0.012,0.031-0.012,0-0.02C-0.664,0.07-0.664,0.074-0.664,0.084z M-0.24,0.895c0-0.003,0.015-0.003,0.015-0.005
    S-0.24,0.888-0.24,0.886l-0.016,0.002C-0.256,0.89-0.256,0.892-0.24,0.895z M-0.725-0.112c0.016-0.01,0.016-0.012-0.046-0.012
    C-0.771-0.122-0.756-0.118-0.725-0.112z M-0.586,0.146c0.015,0,0.031,0.002,0.031,0.002l0.015-0.002
    c0-0.002-0.015-0.002-0.031-0.005C-0.571,0.142-0.571,0.145-0.586,0.146z M-0.063,0.911c0,0,0.016,0,0.016-0.002
    s-0.016-0.004-0.031-0.006h-0.016C-0.079,0.905-0.063,0.907-0.063,0.911z M0.69,0.395C0.69,0.395,0.675,0.396,0.69,0.395
    c0,0.004,0,0.006,0.016,0.008c0,0,0,0,0.015,0C0.706,0.4,0.706,0.396,0.69,0.395z M0.706,0.626L0.706,0.626
    c0.015-0.002,0.015-0.004,0-0.006l0,0C0.706,0.625,0.706,0.625,0.706,0.626z\" " + SvgExporter.SVG_STYLE_HOOK + " />";
        s += "</g>";
        return s;
    }
}

--- a/geometrize/shape/ShapeType.hx
+++ b/geometrize/shape/ShapeType.hx
@@ -12,4 +12,5 @@ package geometrize.shape;
        public var ROTATED_ELLIPSE = 4;
        public var CIRCLE = 5;
        public var LINE = 6;
+       public var BRUSH = 7;
 }

diff --git a/geometrize/shape/ShapeFactory.hx b/geometrize/shape/ShapeFactory.hx
index f16c1b2..5231bcf 100644
--- a/geometrize/shape/ShapeFactory.hx
+++ b/geometrize/shape/ShapeFactory.hx
                        case ROTATED_ELLIPSE:
                               return new RotatedEllipse(xBound, yBound);
+                       case BRUSH:
+                               return new Brush(xBound, yBound);
                        case CIRCLE:

[mastef]

( This worked with geometrize-haxe-web - not sure about other limitations )

[cancerberoSgx]

Aja, good one! As I understand that will impact only the exported SVG but it won't have any impact on the algorithm "evolution" since the rasterization on the bitmap will consider Brush just as a RotatedEllipse. I will try to test it though since it seems straightforward and flexible. Maybe, having the svg configurable as a template makes sense to configure the Brush pattern itself.

(This part is strange though: scale(" + rx + " " + ry + "))

Also I think it's time to consider a more flexible API for custom Shapes, since now the code and enums needs to be modified in order to add new shape types. I would like to test it as an user not having to modify the library. I would propose a new method in runner perhaps or a static one:

var runner = new ImageRunner(...)
runner.registerShapeType(Brush) // fails if there is already a shape registered with the same name
runner.step(shapeTypes: [] // refer to the new type by the name declared on its class, just like the others

@Tw1ddle what do you think?

[Tw1ddle]

Yep, with that code the Brush type will do the same work as a RotatedEllipse, except for changing the SVG visuals (which is what gets displayed as the final image in the geometrize-haxe-web demo). So it's the same as postprocessing the SVG.

It'd be nice to make the SVG export and other functions easier to override though.

Agree that adding custom shapes without changing the library would be useful. Registering custom types (that implement geometrize.Shape? maybe using some reflection) as you suggested should work.

[Sphinxxxx]

Here I have tried the "post-processing" approach. I'm quite happy with how it turned out :)

https://codepen.io/Sphinxxxx/pen/qBRZPeg

[EternityForest]

This is already quite likely the best image-to-painting algorithm that exists in FOSS, the results look more like a real painting than even most of the GAN based stuff.

I still think it should ideally be a part of each step, because some brushes might leave a lot more gaps. I'd like to see what happens with a true brush engine like MyPaint or Krita, maybe you could have a script to generate a few dozen strokes for each of a few sizes for a brush from MyPaint, and compile it into a loadable brush pack?

I can think of a few other ways to really try to get close to museum-grade here:

What if you had an option to do things in multiple stages, which would be somewhat like what a real painter might do?

First you would lay down a low resolution base layer made of high opacity shapes that were pretty large, limited to lighter colors only, then as a separate image you would refine it with some brush strokes with a real painterly brush?

Maybe you could also track paint thickness at every point, to simulate the way light interacts with uneven paint.

The algorithm could consider flatness in the fitness function to a configurable degree, but track the total opacity that's ever been applied to a pixel in the end, and use that with something like GIMP emboss to simulate a height map.

To really refine it, you could also track paint dryness over time, as a brush on wet paint might actually remove some or indent it, and you could get some dripping or bleeding with watercolor, which would be so cool to be able to simulate, and probably not subtle.