search

Sxela / WarpFusion

WarpFusion
Other
964 stars 108 forks source link
img2img stablediffusion text2video vid2vid video video2video

readme

WarpFusion

WarpFusion

Latest public version:

If you find a public version elsewhere, before running as admin/root, make sure to check it for malware by comparing it to the latest notebook in this repo.

Greatly inspired by Cameron Smith's neural-style-tf

Example videos

Example videos and settings

Guides made by users:

Local installation guide for Windows (venv)

Run once per notebook version (recommended)

  1. Create a folder for WarpFusion. It's recommended to have a general folder for WarpFusion and subfolders for each version. Like C:\code\WarpFusion\0.16.11\ for version 0.16.11
  2. Download install.bat and save it into your WarpFolder, C:\code\WarpFusion\0.16.11\ in this example.
  3. Run install.bat. It will download and install python, git, and create a virtual python environment called "env" inside our folder and install dependencies, required to run the notebook and jupyter server for local colab. When git install window appears, use the default settings. The installation will contiinue after you install git.
  4. Download run.bat and save it into your WarpFolder, C:\code\WarpFusion\0.16.11\ in this example.

    Run to launch

  5. Execute run.bat. It will activate the environment and start jupyter server.
  6. After the server has launched, go to https://colab.research.google.com
  7. Click File -> Upload Notebook and upload the *.ipynb file
  8. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  9. Select "connect to a local runtime" and paste the URL that will be generated below, which looks like "http://localhost:8888/?token=somenumbers"
  10. Click "Connect" and CTRL+F9 to run all cells.

Local installation guide for Linux-Ubuntu 22.04 (venv):

Pre-requisites:

Troubleshoot python virtual environment issues

Docker install

Run once to install (and once per notebook version)

  1. Create a folder for warp, for example d:\warp
  2. Download Dockerfile and docker-compose.yml to d:\warp
  3. Edit docker-compose.yml so that volumes point to your model, init_images, images_out folders that are outside of the warp folder. For example, d:\models\:/content/models will expose d:\models as /content/models to the notebook
  4. Download and install docker from here - https://docs.docker.com/get-docker/
  5. Run docker-compose up --build inside the warp folder.
  6. Go to https://colab.research.google.com
  7. Click File -> Upload Notebook and upload the *.ipynb file
  8. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  9. Select "connect to a local runtime" and paste the token that was generated in your docker container, but leave the url as localhost. Should look like "http://localhost:8888/?token=somenumbers"
  10. Click "Connect" and CTRL+F9 to run all cells.

Run to launch

  1. Run docker-compose up inside the warp folder.
  2. Go to https://colab.research.google.com
  3. File -> open notebook -> open your previouslty uploaded notebook
  4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  5. Select "connect to a local runtime" and paste the token that was generated in your docker container, but leave the url as localhost. Should look like "http://localhost:8888/?token=somenumbers"
  6. Click "Connect" and CTRL+F9 to run all cells.

Credits

This notebook uses:

Stable Diffusion by CompVis & StabilityAI\ K-diffusion wrapper by Katherine Crowson\ RAFT model by princeton-vl \ Consistency Checking (legacy) from maua \ Color correction from pengbo-learn\ Auto brightness adjustment from progrockdiffusion

AUTOMATIC1111: weighted prompt keywords, lora, embeddings, attention hacks \ Reconstructed noise - based on changes suggested by briansemrau

ControlNet \ TemporalNet, Controlnet Face and lots of other controlnets (check model list)\ BLIP by SalesForce \ RobustVideoMatting (as external cli package) \ CLIP \ FreeU Hack \ Experimental ffmpeg Deflicker \ Dw pose estimator \ SAMTrack Segment-and-Track-Anything (with cli my wrapper and edits) \ ComfyUI: sdxl controlnet loaders, control loras \ animatediff base \ animatediff wrapper for compvis models from comfyui-animatediff \ IP Adapters implementation from sd-webui-controlnet

\ DiscoDiffusion legacy credits:

Original notebook by Somnai, Adam Letts and lots of other awesome people!

Turbo feature by Chris Allen

Improvements to ability to run on local systems, Windows support, and dependency installation by HostsServer

Warp and custom model support by Alex Spirin

Citation

If you find this code useful for your research, please cite:

@misc{Spirin2022,
  author = {Spirin, Alex},
  title = {warpfusion},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Sxela/WarpFusion}},
}