[1/2] Add Image Ops for Data Augmentation

[abhichou4]

Part 2: https://github.com/tensorflow/addons/issues/1353

As part of #1226, relevant image ops need to be added to tfa.image. List of ops tracked in this issue (tick the op only once the corresponding PR is merged):

• [x] Blend
• [x] CutOut
• [ ] Identity
• [x] Translate X, Y
• [x] Shear X, Y
• [ ] Solarize Ops
• [ ] Posterize
• [ ] AutoContrast
• [x] Equalize
• [ ] Invert
• [ ] Color
• [ ] Brightness
• [x] Sharpness

Reference: https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py

[vinayvr11]

Ok So these unchecked options are remaining. But what is PIL does it stands for Pillow or something else. And i have to implement these in autoaugment.py am i right.

[abhichou4]

Ok So these unchecked options are remaining. But what is PIL does it stands for Pillow or something else. And i have to implement these in autoaugment.py am i right.

That's right, but the appropriate namescapes and where to place these operations is something that can be discussed here or on your PR. /cc @bhack @gabrieldemarmiesse

[vinayvr11]

Ok so for now i just need to make a script for these operations and after that we decide that where to put them all. So all these operations are have to be integrated in one script or i have to make a different scripts for all of them. Thank you

[gabrieldemarmiesse]

Yeah, as long as the public API stays the same and is decided first, it's not critical how you name your files.

[vinayvr11]

@gabrieldemarmiesse : I have some doubts so we have to use the tensorflow for doing these operations or we are free to use any module. And all these methods have to be made in different files. Please correct me if i am wrong.

[gabrieldemarmiesse]

You should always use tensorflow to make those ops (so they can run on tpu/cpu/cuda/ROCm and can be exported in a graph and have gradients and many other things... ) and where you put the ops is not critical (meaning we care more about the pull request being merged than about the name of the file, as it can be changed later without impacting the API). You can group ops together in the same file if you feel this is the right move.

Prefer using python only if possible.

[vinayvr11]

Ok and what about Tensorflow version shall i need to take care about the version also or i can use any version of Tensorflow.

[gabrieldemarmiesse]

We currently test with tensorflow 2.1.0 and 2.2.0rc0

[vinayvr11]

ok thank you

[vinayvr11]

@gabrieldemarmiesse : Do we have a pre defined gaussian filter in tensorflow or any similar to this or i just need to make a filter in order to blur the images.

[bhack]

Just to compare to this ops checkbox list with RemixMatch/FixMatch https://github.com/google-research/remixmatch/blob/master/libml/augment.py#L36-L41

[bhack]

Here the EfficientDet Autoaugment ops list https://github.com/google/automl/blob/master/efficientdet/aug/autoaugment.py#L43-L153

[vinayvr11]

@bhack So do we have to make a new list of these ops or leave them out?

[bhack]

@vinayvr11 I think it could be interesting to make a new list expecially for BBox ops in EfficentDet in a new ticket and instead extend the list here for image ops. But you need to double check the overlapped ops for these lists.

[vinayvr11]

Thank you @bhack i want to know one thing can i use numpy for making some ops.

[bhack]

@vinayvr11 I think the same as https://github.com/tensorflow/addons/issues/1333#issuecomment-600770490

[vinayvr11]

ok so do we have any filter operation in tensorflow_addons or i have to add that from scratch.

[gabrieldemarmiesse]

No idea. You can look in the source code, I don't know all the functions by heart haha

[bhack]

@vinayvr11 We had an Averaging filter already integrated. See also https://github.com/tensorflow/addons/issues/358

[bhack]

/cc @parth-p

[vinayvr11]

Thank you

[bhack]

As Google is a little bit evolutionary fragmented.. :stuck_out_tongue_winking_eye: We have others recent interesting augmentations to be double checked for overlaps in https://github.com/google-research/simclr/blob/master/data_util.py.

There you can find also Tensorflow impl. of Gaussian Blur. /cc @chentingpc

[vinayvr11]

So shall i stop working on this blur ops because i just made a script for blurring using mean_filter.

[abhichou4]

So shall i stop working on this blur ops because i just made a script for blurring using mean_filter.

Use that impl as a reference to add Gaussian Blur.

[vinayvr11]

sure thanks

[bhack]

Can we complete at least the unique ops list in checkbox list for image ops in all these source? Then we could have another new ticket to list all bbox/crop operations.

[vinayvr11]

Thank you for that @abhichou4 i made a gaussian blur script for Tensorflow addons can i send the PR of that.

[bhack]

@abhichou4 Can you edit the ticket to expand the list?

RemixMatch/FixMatch:

• Identity
• AutoContrast
• Equalize
• Rotate
• Solarize
• Color
• Contrast
• Brightness
• Sharpness
• ShearX
• TranslateX
• TranslateY
• Posterize
• ShearY

Then SimCLR..

[vinayvr11]

@bhack : We already have rotate ops in tensorflow_addons -https://www.tensorflow.org/addons/api_docs/python/tfa/image/rotate

[bhack]

Yes I meant to expand over an "unique" list.. If you see we had in the last post also ops already in the original list of the ticket..

[vinayvr11]

Yes so first we need to update the list with unique ops.

[vinayvr11]

I just send an PR for solarize could you please review it.

[vinayvr11]

Hello @abhichou4 i am trying to import the method augment_ops that you added in Addons but it won't show in the files could you please help me to figure this out. Thank you

[abhichou4]

Hello @abhichou4 i am trying to import the method augment_ops that you added in Addons but it won't show in the files could you please help me to figure this out. Thank you

Not merged yet.

[vinayvr11]

Ok so i need this for color can i just assume this and send a PR or wait for merge.

[vinayvr11]

@abhichou4 : Can i use the random image like you generated in your test script or the repository test_data image.

[abhichou4]

@abhichou4 : Can i use the random image like you generated in your test script or the repository test_data image.

test_data seems like a better option. My only concern is dumping all these images with long names. /cc @fsx950223

[fsx950223]

@abhichou4 : Can i use the random image like you generated in your test script or the repository test_data image.

test_data seems like a better option. My only concern is dumping all these images with long names. /cc @fsx950223

I believe test_data is a better option because it could check the actual result.

[vinayvr11]

@abhichou4 : Can i use the random image like you generated in your test script or the repository test_data image.

test_data seems like a better option. My only concern is dumping all these images with long names. /cc @fsx950223

Yes, same here but i only test the operation on one image Yellow_smiley_face.png as this is the smallest one.

[vinayvr11]

I have added the test file as well please check that.

[vinayvr11]

@abhichou4 : posterize is also done please check that.

[vinayvr11]

Could you please merge my commits they have passed all tests.

[bhack]

Just for the note... there are many other interesting augmentations operations that could be inspired by https://github.com/aleju/imgaug (It works also on other datatypes like heatmaps, keypoints etc..). The problem there it is that often it is hard to achieve good perfomance[1][2][3] and it is why we are trying to solve here with TF ops so that we could rely on the TensorFlow compiler/optimization/scheduling/runtime stacks and it is why Nvidia bootstrapped it is using its own library[4]. [1]https://github.com/aleju/imgaug/issues/79 [2]https://github.com/aleju/imgaug/issues/550 [3]https://github.com/aleju/imgaug/issues/178 [4]https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/examples/augmentation_gallery.html

[Squadrick]

@abhichou4 @vinayvr11 @bhack The checklist above should only be ticked once the PR has been merged. I've unselected all the open PRs.

Also, please open a single PR for each image op, even if it's a simple op. It makes it super tough to review when 4/5 ops are combined into a single PR.

@abhichou4 I'm closing #1352 since it's WIP, and implements multiple ops. Create separate issues.

[bhack]

@Squadrick Thanks but I have not the privilege level to tick the checklist :wink:

[seanpmorgan]

Per https://github.com/tensorflow/addons/issues/1226#issuecomment-634755922 the autoaugment/randaugment features will be a part of https://github.com/keras-team/keras-cv

Apologies for not being able to present this information sooner to the contributors.