search

FluxML / DataAugmentation.jl

Flexible data augmentation library for machine and deep learning
https://fluxml.ai/DataAugmentation.jl/dev/
MIT License
41 stars 17 forks source link

Add Normalize path for Image #54

Closed darsnack closed 2 years ago

darsnack commented 2 years ago

This adds a dispatch so that Normalize can be used on an Image (i.e. colorant array). Useful when you need to normalize data then apply augmentations on top of that without converting back and forth.

lorenzoh commented 2 years ago

Hey Kyle, thanks for the addition!

Did you test this for colors that are not stored as Float32? I think there could be some issues with clamping. Also, the normalization stats will probably be floats, so maybe it's best to restrict the argument of the item to Image{N, <:Colorant{<:AbstractFloat}} (or something similar) and require an explicit element type conversion with ToEltype.

Out of curiosity, what is your use case for this?

darsnack commented 2 years ago

I see what you mean about the clipping. The main use case here is when you want to normalize your images, but the augmentations don't compose with the normalization placed at the end of the pipeline. For example, I am doing zero padding before a random crop. If I normalize after the padding, then the pixels will no longer be zero. So either the normalization needs to understand Image or the other augmentations need to understand ArrayItem.

darsnack commented 2 years ago

Okay thinking about this more, I don't think the goal of the PR makes any sense. Even floating point backed colors don't make sense as negative values even if the library implementation doesn't auto-clip things.

I guess in the case where the normalization statistics are provided pre-augmentation, then having variations of RandomCrop, Flip, etc. that accepts a dims argument and applies to ArrayItems would be the appropriate solution.