search

mdbloice / Augmentor

Image augmentation library in Python for machine learning.
https://augmentor.readthedocs.io/en/stable
MIT License
5.06k stars 866 forks source link

Adding operations on keypoints/landmarks? #34

Open carterbao opened 7 years ago

carterbao commented 7 years ago

This tool is awesome, especially for the elastic distortion function. Is is possible to add operations on given keypoints/landmarks, which means the inputs and outputs are images and corresponding points? This is helpful for alignment problems.

mdbloice commented 7 years ago

Hi, thanks for opening the issue. I am not quite sure what you mean. Could you elaborate a little? Maybe with some code of how you'd think such an operation would work with Augmentor or maybe some example input and output images?

carterbao commented 7 years ago

Hi, thank you for replying. For example, if I want to detect key-points(or landmarks) on faces, and I may have some face images with their key-points labeled. Now I want to get more labeled images by image augmenting, for training a neural network. However, after the operation, the key-points will move and I can't get their positions on new images. So is it possible to do the same operation(rotation, affine transformation, etc) to the key-points as well, which means the input could be a pair (input_image, key_points) and the output is a pair (output_image, key_points) ?

Maybe the code will look like this:

import Augmentor
p = Augmentor.Pipeline("/path/to/input/label/file", "/path/to/output/label/file")
p.sample(10000)

and the label file may look like this:

/path/to/image1  key_pt1_x, key_pt1_y, key_pt2_x, key_pt2_y ...
/path/to/image2  key_pt1_x, key_pt1_y, key_pt2_x, key_pt2_y ...
...

so the augmentor reads the key-points with each image, augments the images, and generates an output label file which contains all the generated images and their key-points. Have I made myself clear? You may have a look at this tool: imgaug This tool augments images with basic operations, with key-points preserved. While I love the elastic distortions in your augmentor and wish it could be better. Many thanks if you would consider this.

mdbloice commented 7 years ago

Hi, ok I see what you are saying now. Well currently that won't be possible because I use PIL as a backend for most of the image operations. However, I do plan on replacing PIL as a dependency in a future update, and use only numpy for the image manipulations, which would make it possible to implement what you are asking for. When I do this, I will add this feature. I can't really give you a timeline on when that might be, unfortunately. M.

uilliu commented 5 years ago

This tool is fantastic. I especially like the piping design and randomization of this system. After reading the document, however, I find that I also need to operate the key-points of annotation. I am very much looking forward to adding this feature to Augmentor in the near future, which will be a very important step in data generation.

devforfu commented 5 years ago

Yeah, the tool looks good and I really like the interface. Would be great to have 2d points and bounding boxes augmentation support like you can find in imgaug or fastai libraries. The later one operates with 2d tensors directly.

mdbloice commented 5 years ago

Hi @devforfu, right now I have no plans to add bounding boxes or the like to Augmentor. It may happen in future, but it's not really a priority at the moment. 2D points and bounding boxes would be nice features to have, so I may try to implement them at some point however!