search

herobd / handwriting_line_generation

Code for BMVC2020 paper "Text and Style Conditioned GAN for Generation of Offline Handwriting Lines"
Other
67 stars 29 forks source link

Custom Style Generation #7

Closed eylonmiz22 closed 3 years ago

eylonmiz22 commented 3 years ago

Hey There!

Thank you for your great work, it is incredible.

I was trying to create my own handwriting style vectors by the gen_style.py file, but the results were not as good as the test_styles you have provided in the releases. Actually, it generated handwriting, but the style didn't make sense (it wasn't similar to my handwriting style).

To create the custom style vectors, first I used my own OCR tool to crop the lines from the image I input, then I had to intervene and to change the author_hw_dataset.py a bit, in order to get my cropped lines and manipulate them.

For the generation process, I chose the RANDOM action, but the random sampling was made from my single handwriting style (represented as several styles, each compatible with each line cropped from my own image example).

The commands I used:

  1. python3 get_styles.py -c "$checkpoint_path" -d "/content/new" -g 0 -T -f "$config_path"
  2. !python3 generate.py -c "$checkpoint_path" -d "$test_results_path" -g 0 -T -f "$config_path" -s "/content/new/test_styles_175000.pkl"

"/content/new/test_styles_175000.pkl" is the path of my own line-style vectors, created by command number 1.

Is there anything that could help me to generate handwriting that is more similar to mine?

Again, thank you very much, it is awesome work!

herobd commented 3 years ago

Sorry it wasn't more obvious, but there is a much easier way to generate handwriting based on an example image. generate.py has an f option, which will ask for two image paths (to handwriting line images) and the target text and generate a series of images interpolating between the two styles it extracts. This doesn't save the style vector, but you could modify the code to if you wanted that. You may not want the interpolation, but the first and last images will be 100% the two respective styles. You could even give it the same image twice. If you want to extract the style from several images, you can just append them horizontally (that's how the model does it in training).

It is likely there are two primary reason for poor style matching:

  1. The model doesn't actually mimic a partiular style very well. It mostly mimics global/general things like slant, but isn't good enough to recreate specific character shapes.
  2. Your images are outside of the distribution images it was trained on (IAM). The IAM dataset was captured and preprocessed in a very uniform way. You probably will need to modify your images (contrast, cropping, ink thickness) to resemble the IAM images more closely.
eylonmiz22 commented 3 years ago

Thank you for your time and for your quick reply it really makes sense.

One more question regarding this topic:

I think I already fell through this problem, which was caused by both of the reasons that you've suggested, after training a model called GANWriting and trying to use it to generate my own handwriting. In your great paper, I saw that you may be familiar with the GANWriting paper, as I noticed it was referenced there.

Then, I tried to solve reason number 2 by training shortly the GANWriting pre-trained (on IAM) model on my own handwriting samples. After trying to test the model and to generate my own handwriting given new samples that consist of my handwriting style and distribution, the results were as you mentioned in reason number 1: The model only mimicked global/general features like tilt and thickness and didn't focus on the low-level features as letter shapes, etc.

My question is: Do you think that the same process should be useful, using your model in order to deal with reason number 2 (and not only modifying the images as contrast, cropping, ink thickness)? Does training your pre-trained checkpoint on my own handwriting could surpass the results I gained after I did so using the GANWriting model?

Thanks again!

herobd commented 3 years ago

Theoretically, you should be able to train/fine-tune any generation model on a single author's data and get good reconstruction of that author. The gotcha is you may not have enough data for a given author. Of course, you can always create more examples of your own handwriting. How much data is needed? I don't really know, but I'd imagine 20 lines covering all letters (lowercase and uppercase) multiple times would be enough.

I'm not sure why that wouldn't have worked with GANWriting, unless there weren't enough examples of your handwriting. Making the images look closer to the original data should make it so you need less data.

eylonmiz22 commented 3 years ago

This is very helpful! Thanks for the motivation. In continuation, I may try this and train the model on my own handwriting. I am closing this issue as I have no more questions regarding this topic.