search

aleju / imgaug

Image augmentation for machine learning experiments.
http://imgaug.readthedocs.io
MIT License
14.41k stars 2.45k forks source link

Augmentable for vector fields? #620

Open Erotemic opened 4 years ago

Erotemic commented 4 years ago

Is there an augmentable data structure that currently represents and correctly augments on vector fields (e.g. optical flow)? If not, then consider this issue a feature request.

Consider data where each pixel represents an x,y vector indicating some quantity (e.g. optical flow). The shape of the data would be (H, W, 2). In addition to augmenting the positions of the vectors the orientation would also need to be augmented.

aleju commented 4 years ago

There is no dedicated type for vector fields. I think it has been requested before. Would probably not be that trivial to implement. E.g. how would the vectors be affected by elastic transformations or projections into polar coordinate space.

Erotemic commented 4 years ago

E.g. how would the vectors be affected by elastic transformations or projections into polar coordinate space.

Yeah, that's the main reason why I submitted this as an issue rather than a PR.

I'm not even sure how to define an elastic transform on a vector field, you could simply offset the positions of each vector and leave the orientations unchanged, but that doesn't seem right. There might be a way to do it, but I'd have to think about it for more than the 15 minutes I've given to it.

However, a reasonable solution might be to punt on the difficult transformations and simply raise NotImplementedErrors for now. I'd wager things like elastic transforms are far less commonly used than things like affine transforms, which do have well defined ways to transform these values. Its a sizable task to simply get the base implementation of this in for even the easy transforms, so it might make sense to start with that and then work on handling the harder cases later.