dt.OneOf to generate multiple squares or rectangles

[dcnieho]

Hi! I've been trying to build a image pipeline that drops multiple blobs on an image, and i want each blob to be either a circle or a square. The number of blobs should be random within some range.

I have the below. This works to generate between 4 and 8 either circles or squares, but when i add dt.OneOf to for each blob randomly select between a circle and a square, i get the same circle (same arguments) and same square multiple times, instead of each being a fresh, updated call. What should i change to get this to work?

IMAGE_SIZE = 512
class GrayBackground(dt.Feature):
    # These two are required to make sure that this background can start a pipeline
    __list_merge_strategy__ = dt.MERGE_STRATEGY_APPEND
    __distributed__ = False

    def get(self, image, luminance, **kwargs):
        bg = np.ones((IMAGE_SIZE,IMAGE_SIZE),dtype=np.float64)*luminance
        return bg

class Circle(dt.Feature):
    def get(self, image, position, radius, intensity, **kwargs):
        X, Y = np.meshgrid(np.arange(image.shape[0]), np.arange(image.shape[1]))
        mask = (X - position[0])**2 + (Y - position[1])**2 < radius**2
        image[mask] = intensity
        return image

class Rectangle(dt.Feature):
    def get(self, image, position_tl, width, height, intensity, **kwargs):
        image[int(position_tl[0]):int(position_tl[0]+width), int(position_tl[1]):int(position_tl[1]+height)] = intensity
        return image

image_pipeline = GrayBackground(
      luminance=lambda: np.random.rand()/4
)
circle = Circle(
    position=lambda: np.random.rand(2) * IMAGE_SIZE,
    radius=lambda: 20 + np.random.rand() * 20,
    intensity=1
)
rect = Rectangle(
    position_tl=lambda: np.random.rand(2) * IMAGE_SIZE-100,
    width=lambda: 20 + np.random.rand() * 20,
    height=lambda: 10 + np.random.rand() * 10,
    intensity=.5
)

if False:
    # this works, but only one shape multiple times
    blobs = dt.Repeat(
        feature=circle,
        N=lambda: np.random.randint(4, 8)
    )
else:
    # if for each blob i want to randomly choose which shape, that doesn't work
    # it generates the same arguments for multiple circles or squares. How do i
    # get each to be a unique one?
    blob = dt.OneOf(
        collection=[circle, rect]
    )
    blobs = dt.Repeat(
        feature=blob,
        N=lambda: np.random.randint(4, 8)
    )

image_pipeline >>= blobs

for i in range(5):
    image = image_pipeline.update()()
    plt.imshow(image, cmap='gray', vmin=0, vmax=1)
    plt.show()

I tried using

blob = dt.OneOf(
        collection=lambda:[circle, rect]
    )

and

blob = dt.OneOf(
        collection=[lambda: Circle(
                            position=np.random.rand(2) * IMAGE_SIZE,
                            radius=20 + np.random.rand() * 20,
                            intensity=1
                        ),
                    lambda: Rectangle(
                            position_tl=np.random.rand(2) * IMAGE_SIZE-100,
                            width=20 + np.random.rand() * 20,
                            height=10 + np.random.rand() * 10,
                            intensity=.5
                        )]
    )

And neither worked, either because (of course) functions can't be indexed, or because features can't be lambdas. Thanks!

[HarshithBachimanchi]

Hi!

I can see two possible solutions for this.

1. Firstly, a simpler solution. If you only care about the final image, you can directly generate the blobs of specific shapes in sequence.

image_pipeline = GrayBackground(luminance=lambda: np.random.rand() / 4)

n = 2 # Number of shapes
image_pipeline = image_pipeline >> (circle ^ n) >> (rect ^ n) 

And then call it as usual with update()().

2. The other way is to create a new subclass which can choose a shape randomly (eg., RandomShape) that is derived from superclasses Circle and Rectangle.

import random
class RandomShape(Circle, Rectangle):
    def get(self, image, shape_class=None, shape_params=None, **kwargs):
        if shape_class is None:
            shape_class = random.choice([Circle, Rectangle])
            shape_class.get(self, image, **shape_params.get(shape_class, {}))
        return image

When called, RandomShape will choose either a circle or a rectangle, and the shape specific parameters are extracted from the shape_params dictionary which can be written as follows:

shape_params = {
    Circle: {
        "position": lambda: np.random.rand(2) * IMAGE_SIZE,
        "radius": lambda: 20 + np.random.rand() * 20,
        "intensity": 1,
    },
    Rectangle: {
        "position_tl": lambda: np.random.rand(2) * IMAGE_SIZE - 100,
        "width": lambda: 20 + np.random.rand() * 20,
        "height": lambda: 10 + np.random.rand() * 10,
        "intensity": 0.5,
    },
}

And the rest is straight forward:

image_pipeline = GrayBackground(luminance=lambda: np.random.rand() / 4)
n = 5  # Number of shapes

image_pipeline = image_pipeline >> (RandomShape(shape_params=shape_params) ^ n) 

plt.figure(figsize=(12, 8))
for i in range(12):
    image = image_pipeline.update()()
    plt.subplot(3, 4, i + 1)
    plt.title(str(i))
    plt.imshow(image, cmap="gray", vmin=0, vmax=1)
    plt.axis("off")

Result: Hope this helps!

[dcnieho]

@HarshithBachimanchi Thanks a lot, that worked. Sadly, i now see that my simplified example was too simplified. I want blobs to be fully on my image, so i have dependent properties, where the positioning lambda depends on the radius. This code dies:

import random

IMAGE_SIZE = 512
class GrayBackground(dt.Feature):
    # These two are required to make sure that this background can start a pipeline
    __list_merge_strategy__ = dt.MERGE_STRATEGY_APPEND
    __distributed__ = False

    def get(self, image, luminance, **kwargs):
        bg = np.ones((IMAGE_SIZE,IMAGE_SIZE),dtype=np.float64)*luminance
        return bg

class Circle(dt.Feature):
    def get(self, image, position, radius, intensity, **kwargs):
        X, Y = np.meshgrid(np.arange(image.shape[0]), np.arange(image.shape[1]))
        mask = (X - position[0])**2 + (Y - position[1])**2 < radius**2
        image[mask] = intensity
        return image

class Rectangle(dt.Feature):
    def get(self, image, position_tl, width, height, intensity, **kwargs):
        image[int(position_tl[0]):int(position_tl[0]+width), int(position_tl[1]):int(position_tl[1]+height)] = intensity
        return image

class RandomShape(Circle, Rectangle):
    def get(self, image, shape_class=None, shape_params=None, **kwargs):
        if shape_class is None:
            shape_class = random.choice([Circle, Rectangle])
            shape_class.get(self, image, **shape_params.get(shape_class, {}))
        return image

shape_params = {
    Circle: {
        "radius": lambda: 20 + np.random.rand() * 20,
        "position": lambda radius: np.random.rand(2) * (IMAGE_SIZE-2*radius[0]) + radius[0],
        "intensity": 1,
    },
    Rectangle: {
        "position_tl": lambda: np.random.rand(2) * IMAGE_SIZE - 100,
        "width": lambda: 20 + np.random.rand() * 20,
        "height": lambda: 10 + np.random.rand() * 10,
        "intensity": 0.5,
    },
}
blobs = dt.Repeat(
    feature=RandomShape(shape_params=shape_params),
    N=lambda: np.random.randint(4, 8)
)

image_pipeline = GrayBackground(
      luminance=lambda: np.random.rand()/4
)
image_pipeline >>= blobs

for i in range(5):
    image = image_pipeline.update()()
    plt.imshow(image, cmap='gray', vmin=0, vmax=1)
    plt.show()

Note the line "position": lambda radius: np.random.rand(2) * (IMAGE_SIZE-2*radius[0]) + radius[0], That's the change that causes things to break upon update(). The stacktrace is long, but ends with:

[/usr/local/lib/python3.8/dist-packages/deeptrack/properties.py](https://localhost:8080/#) in <lambda>(_ID)
    104 
    105             # Create the action.
--> 106             return lambda _ID=(): sampling_rule(
    107                 **{key: dep(_ID=_ID) for key, dep in used_dependencies.items()},
    108                 **({"_ID": _ID} if "_ID" in knames else {}),

TypeError: <lambda>() missing 1 required positional argument: 'radius'

Do you know how to make this work? Thanks a lot again!

[dcnieho]

Here's an alternative, but now circles are more likely than rectangles, no good either:

IMAGE_SIZE = 512
class GrayBackground(dt.Feature):
    # These two are required to make sure that this background can start a pipeline
    __list_merge_strategy__ = dt.MERGE_STRATEGY_APPEND
    __distributed__ = False

    def get(self, image, luminance, **kwargs):
        bg = np.ones((IMAGE_SIZE,IMAGE_SIZE),dtype=np.float64)*luminance
        return bg

class Circle(dt.Feature):
    def get(self, image, position, radius, intensity, **kwargs):
        X, Y = np.meshgrid(np.arange(image.shape[0]), np.arange(image.shape[1]))
        mask = (X - position[0])**2 + (Y - position[1])**2 < radius**2
        image[mask] = intensity
        return image

class Rectangle(dt.Feature):
    def get(self, image, position_tl, width, height, intensity, **kwargs):
        image[int(position_tl[0]):int(position_tl[0]+width), int(position_tl[1]):int(position_tl[1]+height)] = intensity
        return image

circle = Circle(
    position=lambda: np.random.rand(2) * IMAGE_SIZE,
    radius=lambda: 20 + np.random.rand() * 20,
    intensity=1
)
rect = Rectangle(
    position_tl=lambda: np.random.rand(2) * IMAGE_SIZE-100,
    width=lambda: 20 + np.random.rand() * 20,
    height=lambda: 10 + np.random.rand() * 10,
    intensity=.5
)

image_pipeline = GrayBackground(
      luminance=lambda: np.random.rand()/4
)
N_BLOBS_RANGE = [1,6]
circs = dt.Repeat(
    feature=circle,
    N=lambda: np.random.randint(N_BLOBS_RANGE[0], N_BLOBS_RANGE[1])
)
image_pipeline >>= circs

rects = dt.Repeat(
    feature=rect,
    N_circ= circs.N,
    N=lambda N_circ: np.random.randint(max(0,N_BLOBS_RANGE[0]-N_circ), N_BLOBS_RANGE[1]-N_circ),
)
image_pipeline >>= rects

plt.figure(figsize=(24, 16))
for i in range(12):
    image = image_pipeline.update()()
    plt.subplot(3, 4, i + 1)
    plt.title(str(i))
    plt.imshow(image, cmap="gray", vmin=0, vmax=1)
    plt.axis("off")

So I guess I'll give up a bit of control over the total number of blobs, and just use:

N_BLOBS_RANGE = [0,3]
circs = dt.Repeat(
    feature=circle,
    N=lambda: np.random.randint(N_BLOBS_RANGE[0], N_BLOBS_RANGE[1])
)
image_pipeline >>= circs

rects = dt.Repeat(
    feature=rect,
    N=lambda: np.random.randint(N_BLOBS_RANGE[0], N_BLOBS_RANGE[1])
)
image_pipeline >>= rects

if the RandomShape solution can't be fixed.

[BenjaminMidtvedt]

Hi!

The correct solution should have been

blobs = dt.OneOf([circle, rect])
blobs ^= lambda: np.random.randint(N_BLOBS_RANGE[0], N_BLOBS_RANGE[1])

However, a small bug made this not work as expected. Theres a PR for it now, will be fixed shortly. In the meantime, I recommend

class GrayBackground(dt.Feature):
    # These two are required to make sure that this background can start a pipeline
    __list_merge_strategy__ = dt.MERGE_STRATEGY_APPEND
    __distributed__ = False

    def get(self, image, luminance, **kwargs):
        bg = np.ones((IMAGE_SIZE, IMAGE_SIZE), dtype=np.float64) * luminance
        return bg

class Circle(dt.Feature):
    def get(self, image, position, radius, intensity, **kwargs):
        X, Y = np.meshgrid(np.arange(image.shape[0]), np.arange(image.shape[1]))
        mask = (X - position[0]) ** 2 + (Y - position[1]) ** 2 < radius**2
        image[mask] = intensity
        return image

class Rectangle(dt.Feature):
    def get(self, image, position_tl, width, height, intensity, **kwargs):
        image[
            int(position_tl[0]) : int(position_tl[0] + width),
            int(position_tl[1]) : int(position_tl[1] + height),
        ] = intensity
        return image

class RandomShape(Circle, Rectangle):
    def __init__(self, features, **kwargs):
        super().__init__(**kwargs)
        self.features = features
        for feature in features:
            # This adds the feature to the dependency graph
            # so it gets updated correctly by .update()
            self.add_feature(feature)

    def get(self, image, **kwargs):
        feature = random.choice(self.features)
        return feature(image, **kwargs)

circle = Circle(
    position=lambda: np.random.rand(2) * IMAGE_SIZE,
    radius=lambda: 20 + np.random.rand() * 20,
    intensity=1,
)
rect = Rectangle(
    position_tl=lambda: np.random.rand(2) * IMAGE_SIZE - 100,
    width=lambda: 20 + np.random.rand() * 20,
    height=lambda: 10 + np.random.rand() * 10,
    intensity=0.5,
)

image_pipeline = GrayBackground(luminance=lambda: np.random.rand() / 4)

shape = RandomShape([circle, rect])
image_pipeline >>= shape ^ (
    lambda: np.random.randint(N_BLOBS_RANGE[0], N_BLOBS_RANGE[1])
)

Also, for future reference, GrayBackground can be replaced with dt.Value

[dcnieho]

@BenjaminMidtvedt Super, thanks! That works well now indeed. Nice concise code when making use of the ^= operator too.

Thanks also for the tip about dt.Value. I've now made it into:

image_pipeline = dt.Value(
      luminance=lambda: np.random.rand() / 4,
      value=lambda luminance: np.ones((IMAGE_SIZE,IMAGE_SIZE),dtype=np.float64)*luminance
)

Is that what you meant (and i have the separate luminance parameter just for logging)?

[BenjaminMidtvedt]

Yep, exactly!