TouchDiffusion

TouchDesigner implementation for real-time Stable Diffusion interactive generation with StreamDiffusion.

Benchmarks with stabilityai/sd-turbo, 512x512 and 1 batch size.

Disclaimer

Notice: This repository is in an early testing phase and may undergo significant changes. Use it at your own risk.

Usage

[!TIP] TouchDiffusion can be installed in multiple ways. Portable version have prebuild dendencies, so it prefered way to install or Manuall install is step by step instruction.

Portable version:

Includes preinstalled configurations, ensuring everything is readily available for immediate use.

1. Download and extract archive
2. Run webui.bat. It will provide url to web interface (ex. http://127.0.0.1:7860)
3. Open install & update tab and run Update dependencies.

Manuall install:

You can follow YouTube tutorial

Required TouchDesigner 2023 & Python 3.11

1. Install Python 3.11
2. Install Git
3. Install CUDA Toolkit 11.8 (required PC restart)
4. Download TouchDiffusion.
5. Open webui.bat with text editor and set path to Python 3.11 in set PYTHON_PATH=. (ex. set PYTHON_PATH="C:\Program Files\Python311\python.exe")
6. Run webui.bat. After installation it will provide url to web interface (ex. http://127.0.0.1:7860)
7. Open install & update tab and run Update dependencies. (could take ~10 minutes, depending on your internet connection)
8. If you get pop up window with error related to .dll, run Fix pop up
9. Restart webui.bat

Accelerate model:

Models in .safetensors format must be in models\checkpoints folder. (as for sd_turbo, it will be auto-downloaded).

Internet connection required, while making engines.

1) Run webui.bat 2) Select model type. 3) Select model. 4) Set width, height and amount of sampling steps (Batch size) 5) Select acceleration lora if available. 6) Run Make engine and wait for acceleration to finish. (could take ~10 minutes, depending on your hardware)

TouchDesigner inference:

1. Add TouchDiffusion.tox to project
2. On Settings page change path to TouchDiffusion folder (same as where webui.bat).
3. Save and restart TouchDesigner project.
4. On Settings page select Engine and click Load Engine.
5. Connect animated TOP to input. Component cook only if input updates.

Known issues / Roadmap:

• [x] Fix Re-init. Sometimes required to restart TouchDesigner for initializing site-packages.
• [ ] Code clean-up and rework.
• [x] Custom resolution (for now fixed 512x512)
• [ ] CFG not affecting image
• [ ] Add Lora
• [x] Add Hyper Lora support
• [ ] Add ControlNet support
• [ ] Add SDXL support

Acknowledgement

Based on the following projects:

• StreamDiffusion - Pipeline-Level Solution for Real-Time Interactive Generation
• TopArray - Interaction between Python/PyTorch tensor operations and TouchDesigner TOPs.