How to get a Processable input svg file for CLIPFont?

[wendashi]

Thanks for your awesome job❤️ I reproduced the CLIPFont.

But when I use new input svg file, I meet some Error. I thought maybe I need to process the svg into a certain format? Could you please give me some advice? Thanks a lot🙏

1. input bash: python svg_render.py x=clipfont prompt='Starry Night by Vincent van gogh' target='./data/test1.svg'
2. input svg:
3. errors:

python svg_render.py x=clipfont prompt='Starry Night by Vincent van gogh' target='./data/test1.svg'
==> system args: 
{'target': './data/test1.svg', 'prompt': 'Starry Night by Vincent van gogh', 'neg_prompt': None, 'state': {'cpu': False, 'mprec': 'no'}, 'diffuser': {'download': False, 'force_download': False, 'resume_download': False}, 'diffvg': {'print_timing': False}, 'seed': 951222, 'multirun': False, 'srange': None, 'result_path': './workspace', 'save_step': 10, 'eval_step': 10, 'mv': False, 'framefreq': 5, 'framerate': 24, 'output_dir': '/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/workspace/clipfont-2024-04-23-14-16'}
==> yaml config args: 
{'method': 'clipfont', 'lr_base': {'point': 0.1, 'color': 0.01}, 'lr_decay_rate': 0.1, 'decay_steps': [1000, 1500], 'lr_schedule': False, 'num_iter': 200, 'batch_size': 1, 'font': {'reinit': False, 'reinit_color': 'randn'}, 'clip': {'model_name': 'ViT-B/32'}, 'thresh': 0.0, 'num_crops': 128, 'crop_size': 230, 'lam_patch': 150, 'lam_dir': 30, 'lam_lpips': 0, 'lam_l2': 0}

***** Model State *****
-> Mixed Precision: no, AMP: False
-> Weight dtype:  torch.float32
-> Working Space: '/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/workspace/clipfont-2024-04-23-14-16/sd951222-lpips0-l20'
Process 0 using device: cuda
-> state initialization complete 

-> init svg from `/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/workspace/clipfont-2024-04-23-14-16/sd951222-lpips0-l20/test1_scale.svg` ...
init_image shape:  torch.Size([1, 3, 512, 512])
Error executing job with overrides: ['x=clipfont', 'prompt=Starry Night by Vincent van gogh', 'target=./data/test1.svg']
Traceback (most recent call last):
  File "/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/svg_render.py", line 143, in <module>
    main()
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/stone/miniconda3/envs/svgrender/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/svg_render.py", line 113, in main
    pipe.painterly_rendering(svg_path=cfg.target, prompt=cfg.prompt)
  File "/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/pytorch_svgrender/pipelines/CLIPFont_pipeline.py", line 108, in painterly_rendering
    optimizer.init_optimizers()
  File "/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/pytorch_svgrender/painter/clipfont/painter_params.py", line 122, in init_optimizers
    self.renderer.set_parameters()
  File "/home/stone/Desktop/AnyFont/svgrender/PyTorch-SVGRender/pytorch_svgrender/painter/clipfont/painter_params.py", line 81, in set_parameters
    path.points.requires_grad = True
AttributeError: 'Rect' object has no attribute 'points'

[wendashi]

I found some info from svgrender/PyTorch-SVGRender/data/alphabet1.svg. But I'm not sure if every svg file from "Adobe Illustrator 25.0.1, SVG Export Plug-In" can be the inputs of CLIPFont?

<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 25.0.1, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
<svg version="1.1" id="图层_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
     viewBox="0 0 512 512" style="enable-background:new 0 0 512 512;" xml:space="preserve">
<style type="text/css">
    .st0{fill:#FFFFFF;}

[wendashi]

May I use some python script or library to process the .ttf into svg which is suitable for CLIPFont? 🤔

[ximinng]

Dear wendashi,

I'm not sure, but it's possible that the output from Adobe Illustrator contains tags that are not supported in DiffVG. Yes, I suggest using FreeType to process .ttf files into SVG.

[ximinng]

Additionally, you can try the Word-As-Img pipeline in PyTorch-SVGRender, which will allow you to obtain glyphs in vector format.

[wendashi]

Dear ximing,

Thanks for your advice❤️.And I found a pixels2svg (https://pypi.org/project/pixels2svg/) can process a 512 x 512 png into the svg format which is suitable for CLIPFont.

But I guess it's still need extra process for more colorful results like the pics in the CLIPFont paper. In the paper, it mentioned the enhancement of original svg as following:

"Specifically, for each vector polygon, increase the number of control points on the boundary. In addition, vector polygon layers are stacked in the channel dimension to expand the parameter search space. Last but not the least, we apply a random color gradient to the input monochromatic font and divide it into multiple small polygons by color similarity. "

Do you have any advice for it? Thanks a lot🙏 And If I can reproduce it, I also can contribute the code to svgrender.

[wendashi]

Additionally, I have tried the Word-As-Img pipeline, but I think it's only suitable for English letters? Cause it defines a rule of control points for each English letters in '/path/to/svgrender/PyTorch-SVGRender/pytorch_svgrender/painter/wordasimage/painter_params.py'. 🤔

 def preprocess_font(self, word, letter, level_of_cc=1, font_path=None, init_path=None):
        if level_of_cc == 0:
            target_cp = None
        else:
            target_cp = {"A": 120, "B": 120, "C": 100, "D": 100,
                         "E": 120, "F": 120, "G": 120, "H": 120,
                         "I": 35, "J": 80, "K": 100, "L": 80,
                         "M": 100, "N": 100, "O": 100, "P": 120,
                         "Q": 120, "R": 130, "S": 110, "T": 90,
                         "U": 100, "V": 100, "W": 100, "X": 130,
                         "Y": 120, "Z": 120,
                         "a": 120, "b": 120, "c": 100, "d": 100,
                         "e": 120, "f": 120, "g": 120, "h": 120,
                         "i": 35, "j": 80, "k": 100, "l": 80,
                         "m": 100, "n": 100, "o": 100, "p": 120,
                         "q": 120, "r": 130, "s": 110, "t": 90,
                         "u": 100, "v": 100, "w": 100, "x": 130,
                         "y": 120, "z": 120}
            target_cp = {k: v * level_of_cc for k, v in target_cp.items()}

[ximinng]

Dear wendashi,

Note that the SVG path obtained through pixel2svg does not contain the fill_color parameter, because this is the color parameter optimized by clipfont. In addition, it is basically the same with word-as-img about the boundary control of detailed graphics (The part of the code you show). word-as-img currently manually defines the number of control points for English letters only. I think for Chinese, the situation is more complicated.

best regards, ximing

[wendashi]

Dear ximing,

Thanks for your advice, I think I make it. Just convert the black words SVG into PNG, and write a python script to fill the PNG color with gird, and then convert the colorful PNG back to SVG.