= Animation Stylization Collection
:imagesdir: asset
:tip-caption: 💡
:note-caption: 📝
:warning-caption: ⚠️
:caution-caption: 🔔
:important-caption: ❗
:experimental:
:toc:
== Environment
[source,bash]
conda create -n torch python=3.8
conda activate torch
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch
pip install pytorch-lightning==1.0.2 opencv-python matplotlib joblib scikit-image torchsummary webdataset albumentations more_itertools
== Algorithm
=== https://github.com/TachibanaYoshino/AnimeGANv2[AnimeGAN: A Novel Lightweight GAN for Photo Animation]
[cols="^.^5,<.^50,<.^50"]
|===
a| Setup
2+a|
. [yellow]#If you just want to use it, just skip the following step#
. download dataset from https://github.com/TachibanaYoshino/AnimeGAN/releases/tag/dataset-1[here] and unzip
. download pretrain VGG19 from https://drive.google.com/file/d/1j0jDENjdwxCDb36meP6-u5xDBzmKBOjJ/view?usp=sharing[here] and unzip, then put it to models/vgg19.npy
|Train
2+a|
[source,bash]
make train CODE=scripts/animegan_pretrain.py CFG=configs/animegan_pretrain.yaml
- training generator (use kbd:[Ctrl+c] can stop)
[source,bash]
make train CODE=scripts/animeganv2.py CFG=configs/animeganv2.yaml
[source,bash]
make tensorboard LOGDIR=logs/animeganv2/
|test
2+a|
. you can download my pretrained model from https://drive.google.com/drive/folders/1Bu5yIYBPGBlO4yNzUamhWdWs5o5gT1Rx?usp=sharing[here].
. run test command
[source,bash]
make infer CODE=scripts/animeganv2.py \
CKPT=logs/animeganv2/version_0/checkpoints/epoch=17.ckpt \
EXTRA=image_path:asset/animegan_test2.jpg # <1>
make infer CODE=scripts/animeganv2.py \
CKPT=logs/animeganv2/version_0/checkpoints/epoch=17.ckpt \
EXTRA=help # <2>
<1> The `EXTRA` paramter can pass `image path` or `image directory path`
<2> Read help.
.2+| Result
a| image::animegan_test2.jpg[animegantest,pdfwidth=50%,scaledwidth=50%]
a| image::animegan_test2_out.jpg[animegantestout,pdfwidth=50%,scaledwidth=50%]
2+^.^a| image::anime_video.png[animevideo,624,432,pdfwidth=50%,scaledwidth=50%,link="https://www.bilibili.com/video/BV1gy4y1z7o2"]
|===
[NOTE]
====
. If you want to change style, you have run `scripts/animegan_datamean.py` get dataset mean difference, then change `configs/animeganv2.yaml` menu:dataset[data_mean].
====
=== https://github.com/SystemErrorWang/White-box-Cartoonization[Learning to Cartoonize Using White-box Cartoon Representations]
[cols="^.^5,<.^50,<.^50"]
|===
a| Setup
2+a|
. [yellow]#If you just want to use it, **just skip** the following step#
. download dataset from https://drive.google.com/file/d/10SGv_kbYhVLIC2hLlz2GBkHGAo0nec-3/view[here] and unzip
. download pretrain VGG19 from https://drive.google.com/file/d/1j0jDENjdwxCDb36meP6-u5xDBzmKBOjJ/view?usp=sharing[here] and unzip, then put it to `models/vgg19.npy`
|Train
2+a|
* change **configs/whitebox_pretrain.yaml** menu:dataset[root] to **your path**
* change **configs/whitebox.yaml** menu:dataset[root] to **your path**
* pre-training generator
[source,bash]
----
make train CODE=scripts/whiteboxgan_pretrain.py CFG=configs/whitebox_pretrain.yaml
----
* training generator (use kbd:[Ctrl+c] can stop)
[source,bash]
----
make train CODE=scripts/whiteboxgan.py CFG=configs/whitebox.yaml
----
* check progress
[source,bash]
----
make tensorboard LOGDIR=logs/whitebox
----
|test
2+a|
. you can download my pretrained model from https://drive.google.com/drive/folders/1Bu5yIYBPGBlO4yNzUamhWdWs5o5gT1Rx?usp=sharing[here]. you can use choice `whitebox-v2.zip` or `whitebox.zip`
. run test command
[source,bash]
----
make infer CODE=scripts/whiteboxgan.py \
CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt \
EXTRA=image_path:asset/whitebox_test.jpg # <1>
make infer CODE=scripts/whiteboxgan.py \
CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt \
EXTRA=image_path:tests/test.flv,device:cuda,batch_size:4 # <2>
# ffmpeg -i xx.mp4 -vcodec libx265 -crf 28 xxx.mp4
make infer CODE=scripts/whiteboxgan.py \
CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt \
EXTRA=help # <3>
----
<1> The `EXTRA` paramter can pass `image path` or `image directory path`
<2> Using GPU convert video, if GPU ran out of memory, pleas use `cpu`
<3> Read help.
.2+| Result
a| image::whitebox_test.jpg[whiteboxtest,pdfwidth=50%,scaledwidth=50%]
a| image::whitebox_test_out.jpg[whiteboxtestout,pdfwidth=50%,scaledwidth=50%]
2+^.^a| image::whitebox_video.png[whitebox_video,pdfwidth=50%,scaledwidth=50%,link="https://www.bilibili.com/video/BV14V411a73o/"]
|===
[NOTE]
====
. The menu:model[superpixel_fn] has a great influence on the style. The `slic,sscolor` refer from offical code. defualt use `sscolor`.
. Pytorch version and official version compare results in https://github.com/zhen8838/AnimeStylized/issues/2#issuecomment-726512435[here]
====
=== https://github.com/minivision-ai/photo2cartoon[Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation(minivision)]
[cols="^.^5,<.^50,<.^50"]
|===
a| Setup
2+a|
. [yellow]#If you just want to use it, **just skip** the following step#
. download dataset from https://drive.google.com/file/d/10SGv_kbYhVLIC2hLlz2GBkHGAo0nec-3/view[here] and unzip
. download `model_mobilefacenet.pth` from https://drive.google.com/file/d/10SGv_kbYhVLIC2hLlz2GBkHGAo0nec-3/view[here] and put into `./models`
|Train
2+a|
* change **configs/uagtit.yaml** menu:dataset[root] to **your path**
* training generator (use kbd:[Ctrl+c] can stop)
[source,bash]
----
make train CODE=scripts/uagtit.py CFG=configs/uagtit.yaml
----
* check progress
[source,bash]
----
make tensorboard LOGDIR=logs/uagtit
----
|test
2+a|
. you can download my pretrained model from https://drive.google.com/drive/folders/1Bu5yIYBPGBlO4yNzUamhWdWs5o5gT1Rx?usp=sharing[here]. (I have not enough GPU and time, so this model effect not so good)
. this model requires an input image that only contains the human head. Since I don’t have time to migrate the previous tools to PyTorch, need to rely on the previous library to process images.(you need clone https://github.com/zhen8838/playground[this repo])
[source,bash]
----
python tools/face_crop_and_mask.py \
--data_path test/model_image \
--save_path test/model_image_faces \
--use_face_crop True \
--use_face_algin False \
--face_crop_ratio 1.3
----
. run test command
[source,bash]
----
make infer CODE=scripts/uagtit.py \
CKPT=logs/uagtit/version_13/checkpoints/epoch=15.ckpt \
EXTRA=image_path:asset/uagtit_test.png
----
.2+| Result
a| image::uagtit_test.png[uagtit_test,pdfwidth=50%,scaledwidth=50%]
a| image::uagtit_test_out.png[uagtit_testout,pdfwidth=50%,scaledwidth=50%]
|===
[NOTE]
====
. The menu:model[light] control the model version, the light version requires GPU memory > 8G,non-light version requires GPU memory > 22G.
. Maybe you need training more epoch to get better results.
====
== Repository structure
[%autowidth,cols="<.^,<.^"]
|===
| **Path** | **Description**
| AnimeStylized | Repository root folder
| ├ asset | Folder containing readme image assets
| ├ <
> a| Folder containing configs defining model/data paramters, training hyperparamters.
| ├ <> a| Folder with various dataset objects and transfroms.
| ├ losses | Folder containing various loss functions for training, Only very general used loss functions are added here.
| ├ <> a| Folder containing all the models and training objects
| ├ optimizers | Folder with common used optimizers
| ├ <> a| Folder with running scripts for training and inference
| ├ utils | Folder with various utility functions
|===
[NOTE]
====
[[anchor-configs]]configs::
* Each algorithm has a corresponding config file.
* Config file uses the YAML format
[[anchor-datamodules]]datamodules::
* The `dataset.py`,`dsfunction.py`,`dstransform.py` contains common data module object's basic component
* Basically, each algorithm has a corresponding `xxxds.py`
[[anchor-models]]models::
* Basically, each algorithm has a corresponding `xxxnet.py`
* Now, only have the gan architecture model, in future maybe add more.
[[anchor-scripts]]scripts::
* Each algorithm has a corresponding `xxx.py`, implement the main training step and inference here
* Each algorithm must add `run_common(xxxModule, xxxDataModule)` in main function, then you can use general trainer to training or testing
====
== Participation and Contribution
. Add custom `LightningDataModule` object as `xxxds.py` in `datamodules` dir.
. Add custom `Module` object model architecture as `xxxnet.py` in `networks` dir.
. Add custom `LightningDataModule` training script as `xxx.py` in `scripts` dir
. Add config file in `configs` dir, the paramters follow your custom `LightningModule` and `LightningDataModule`
. trianing your algorithm