Newspaper Augmentation, Dithering / Halftones Effect

[jboarman]

A new halftone or dithering effect should affect colors and greys more than blacks, which means that text should not get affected by this effect to the same degree as images.

• https://github.com/Utkarsh-Deshmukh/image-dithering-python
• https://github.com/hbldh/hitherdither
• https://github.com/philgyford/python-halftone

Great reference on dithering (also has some strange VB code at the end that might be informative, maybe). https://tannerhelland.com/2012/12/28/dithering-eleven-algorithms-source-code.html

[kwcckw]

@proofconstruction are you working on this?

[jboarman]

@kwcckw Feel free to create a notebook using these libraries with a demo image that has text and image like this one which you could wget into the notebook from the same URL of this github-hosted image below. Once we have a proof-of-concept, then creating the aug should be simple.

I have a feeling that we may want to randomize the various dithering and halftone options, so that should be interesting to demo the various possibilities.

[kwcckw]

@kwcckw Feel free to create a notebook using these libraries with a demo image ...

Okay, initially i was checking with @proofconstruction since the assignee is @proofconstruction. It would be redundant if both of us are working on the same thing at once. Anyway, i will post again when i completed the poc in the notebook.

[proofconstruction]

I haven't begun implementing this yet, so go ahead and do it if you like!

For this effect, it should be possible to use the dimensions of the input image to create a mask that tiles the area with circles of some small radius, then subtract this mask from a full np.ones array to get a "negative" image of the circles. This mask can then be applied over the ink layer to produce an image where the ink is made of these small circles.

[kwcckw]

From my initial trial, i'm using Floyd–Steinberg dithering: https://en.wikipedia.org/wiki/Floyd%E2%80%93Steinberg_dithering

And here's the output:

But the correctness of algorithm itself is important? Or any algorithm would be fine as long we are able to produce the visual effect?

[jboarman]

When it comes to training on the output that Augraphy might create, "correctness" is purely a matter of being able to reproduce the kind of content you see in the real world. Having a large variety of output or processing techniques is also helpful since the real world seems to frequently present with unexpected scenarios that are not handled by a single method.

Dithering is one thing, and the dotted half-tones commonly seen in (old) newspaper print is another thing. But both techniques are useful for helping to create datasets that will allow a deep neural network to generalize on a solution.

So, it's important that we reproduce the halftone effect that you see in newspapers for this augmentation to be useful for that purpose. It may take some study of some of the photoshop "HOW TOs" online that show step-by-step how someone using photoshop creates these effects. But, the simplest place to start for now is to create a notebook that uses all 3 of the referenced libraries (and others if needed) so that we can inspect the results for how these images are processed using these libraries. Then, we can make a more informed decision on which techniques are worth pursuing for inclusion in Augraphy.

Side Note: If you find the libraries don't handle color natively, then I think it should be as simple as first splitting the 3 channels of a color image, processing each channel separately, then merging the 3 channels back into 1 images. Does that sound right?

[kwcckw]

So, it's important that we reproduce the halftone effect that you see in newspapers for this augmentation to be useful for that purpose. It may take some study of some of the photoshop "HOW TOs" online that show step-by-step how someone using photoshop creates these effects. But, the simplest place to start for now is to create a notebook that uses all 3 of the referenced libraries (and others if needed) so that we can inspect the results for how these images are processed using these libraries. Then, we can make a more informed decision on which techniques are worth pursuing for inclusion in Augraphy.

Okay, probably that's the best way now since sometimes a visually attractive output might not contributing to a better result in deep learning process, so i guess we can include all of those methods and let the user try it.

Side Note: If you find the libraries don't handle color natively, then I think it should be as simple as first splitting the 3 channels of a color image, processing each channel separately, then merging the 3 channels back into 1 images. Does that sound right?

Yea, we can do that, but I'm not really sure on the 'correctness' of the output:

What do you think about the output above?

[jboarman]

Can you share a notebook where we can preview all the variations?

It seems clear that this will take quite a bit of experimentation and going back-and-forth here will probably be too draining! :stuck_out_tongue:

[kwcckw]

Can you share a notebook where we can preview all the variations?

It seems clear that this will take quite a bit of experimentation and going back-and-forth here will probably be too draining! 😛

Sure, I will share the notebook again once I drafted those methods.

[kwcckw]

I added the illustrations here: https://colab.research.google.com/drive/1ri957-GvdClt2oMQotVPo46u_nV-3gTT?usp=sharing

At this point, it covers:

1. Floyd–Steinberg dithering
2. Random dithering
3. Average dithering
4. Ordered dithering based on bayer method
5. Halftone (support grayscale for now)

I rewrite all of those methods since we would need some manual parameters adjustment later. For each of the dithering method, I'm trying to apply it differently by using different colour space and image channels. I tried to apply it on their grayscale image, rgb channel image, hsv channel image, hsv-h channel, hsv-s channel and hsv-v channel, and looks like some of them yield quite unique result.

[kwcckw]

So any comment on those outputs? Otherwise i will proceed to add them into augmentation and create a pr request.

[proofconstruction]

Getting to this one really late, I'm sorry.

I think the Floyd-Steinberg and ordered dithering in the grayscale channel look great, and you could submit a PR for them. Maybe take the dither_Floyd_Steinberg and dither_order functions you already have and wrap them in a Dither class, which takes a string algorithm in the constructor and if self.algorithm == "ordered" runs the ordered dither, and otherwise runs Floyd-Steinberg on the input image.

I think we should also have the option to use this in the ink or post phases.

[kwcckw]

Sure, i will add this in the next update.

[proofconstruction]

This commit added support for dithering.