Readme for version 0.4.4 and earlier: Readme v0.4.4
Training script and Flownet4 tested in production on Linux
Flownet4 tested on MacOS Apple Silicon with Pytorch 2.3.1
(done) Add generalization logic to training script
(done) Refine batch retime script
(done) Share pre-trained weights for Flownet4 to enable using it in prod with command line and limited interface.
Optimize memory usage for Flownet4
Add Fluidmorph and Interpolate logic to new Flame script.
Test on Intel Mac with Pytorch 2.2.2 (Latest version avaliable for x86 Macs)
cat flameTimewarpML_v0.4.5-dev003.Linux.tar.gz.part-* > flameTimewarpML_v0.4.5-dev003.Linux.tar.gz
tar -xvf flameTimewarpML_v0.4.5-dev003.Linux.tar.gz
pre-configured miniconda environment should be placed into hidden "packages/.miniconda" folder
the folder is hidden (starts with ".") in order to keep Flame from scanning it looking for python hooks
pre-configured python environment usually packed with release tar file
download Miniconda for Mac or Linux (I'm using python 3.11 for tests) from https://docs.anaconda.com/free/miniconda/miniconda-other-installer-links/
install downloaded Miniconda python distribution, use "-p" to select install location. For example:
sh ~/Downloads/Miniconda3-py311_24.1.2-0-Linux-x86_64.sh -bfsm -p ~/miniconda3
eval "$(~/miniconda3/bin/conda shell.bash hook)"
conda create --name appenv --clone base
conda activate appenv
conda install pyqt
conda install numpy
conda install conda-pack
Install pytorch. Please look up exact commands depending on OS and Cuda versions
Linux example
conda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia
MacOS example:
conda install pytorch::pytorch torchvision -c pytorch
Install rest of the dependencies
pip install -r {flameTimewarpML folder}/requirements.txt
Pack append environment into a portable tar file
conda pack --ignore-missing-files -n appenv
mkdir {flameTimewarpML folder}/packages/.miniconda/appenv/
tar xvf appenv.tar.gz -C {flameTimewarpML folder}/packages/.miniconda/appenv/
rm appenv.tar.gz
Finetune option is avaliable as a menu item starting from 0.4.5 dev 003
Export as Linear ACEScg (AP1) Uncompressed OpenEXR sequence. Export your shots so each shot are in separate folder. Copy pre-trained model file (large or lite) to the file to train with. If motion is fast place the whole shot or its parts with fast motion to "fast" folder.
cd {flameTimewarpML folder}
./train.sh --state_file {Path to copied state file}/MyShot001.pth --generalize 1 --lr 4e-6 --acescc 0 --onecycle 1000 {Path to shot}/{fast}/
cd {flameTimewarpML folder}
./train.sh --state_file {Path to MyModel}/MyModel001.pth --model flownet4_v004 --batch_size 4 {Path to Dataset}/
The batch size and learning rate are two crucial hyperparameters that significantly affect the training process and empirical tuning is necessary here. When the batch size is increased, the learning rate can often be increased as well. A common heuristic is the linear scaling rule: when you multiply the batch size by a factor k, you can also multiply the learning rate by k. Another approach is the square root scaling rule: when you multiply the batch size by k multiply the learning rate by sqrt(k)
Training script will scan all the folders under a given path and will compose training samples out of folders where .exr files are found. Only Uncompressed OpenEXR files are supported at the moment.
Export your shots so each shot are in separate folder. There are two magic words in shot path: "fast" and "slow" When "fast" is in path 3-frame window will be used for those shots. When "slow" is in path 9-frame window will be used. Default window size is 5 frames. Put shots with fast motion in "fast" folder and shots where motion are slow and continious to "slow" folder to let the model learn more intermediae ratios.
Samples for training are 3 frames and ratio. The model is given the first and the last frame and tries to re-create middle frame at given ratio.
[TODO] - add documentation