search

mdbloice / Augmentor

Image augmentation library in Python for machine learning.
https://augmentor.readthedocs.io/en/stable
MIT License
5.06k stars 866 forks source link

How to augment images and their (multiple) masks simultaneously, using masks as labels for Keras model? #184

Open gholste opened 5 years ago

gholste commented 5 years ago

Hi,

I'm not sure I've seen a way to use this package for multi-class semantic segmentation. I am aware that using DataPipeline, I can accomplish the first part of this question (augment images and their masks in parallel). However, I don't know how I could train a Keras model with this generator, denoting the masks as labels.

I think the nicest solution to this would be to simply allow the "labels" argument of keras_generator_from_array to also be an "array-like matrix" just like the "images" argument. The function would then apply the same operations to "images" and "labels." This would be immensely helpful, as Keras does not deal with multiple masks at once very nicely.

Even allowing the "labels" argument to be optional would be helpful (for my task, a whole-image label does not make sense). Then I could create two Keras generators from the same Pipeline (one for input images and one for masks), then zip them and feed that to Keras' fit_generator.

VincentYCYao commented 5 years ago

Issue 1: Interpolation of mask after random rotation I have tried augmenting images and the corresponding mask in parallel. After random rotation, intermediate values appear in the binary mask probably due to interpolation. There is no API for the user to choose Nearest Neighbor Interpolation for augmenting masks. I think that would be useful. Though, it is not a big issue, it is something the user should be aware of. random flipping does not have this issue random rotation has it

Issue 2: Severe misalignment between image and mask after random distortion This is a bug

def augmentation(x_root, y_root):
    p = Augmentor.Pipeline(x_root)
    p.ground_truth(y_root)

    # TODO: Bug fix: Augmentor can not synchronize the random distortion between image and label
    p.random_distortion(probability=1, grid_width=4, grid_height=4, magnitude=8)

    # save augmented images
    num_of_samples = int(1e2)
    p.sample(num_of_samples)

distored_mask distore_image

mdbloice commented 5 years ago

HI @holste1 - I will work on something to be able to do what you mentioned for the next version. Recently I have not had much time for Augmentor development, sorry for the delay in replying. I have marked this issue as an enhancement so it will get priority.