search

mattorp / synesthesia-ecosystem

This is a place to share, discuss, request, track, and provide feedback for community efforts for Synesthesia.
MIT License
13 stars 3 forks source link

Style transfer #21

Open mattorp opened 3 years ago

mattorp commented 3 years ago

TJackal:

What’s the ETA on the style transfer software upgrade? I keep going back to https://youtu.be/I0zG9CYufFQ and it’d be an awesome way to get a leg up on Resolume and TouchDesigner seeing as neither have style transfer capabilities as far as I’m aware.

https://discord.com/channels/729824270913503374/752022203205287936/901068476578414592

Examples of implementation:

naprigg commented 3 years ago

More specifically, Dan Treiman has a working style transfer implementation that would allow features that (as far as I'm aware) one would need multiple programs for - see https://github.com/dantreiman/neural-style-tf . While his CLI is well thought-out, a lot of VJs are not expert coders. The primary goal of this feature would be a GUI with the following:

  1. Sections for the images we intend to transfer the style of, as well as our intended video clip
  2. Depth sliders to control weight of https://github.com/dantreiman/neural-style-tf#content--style-tradeoff to allow the mentioned interpolation
  3. A selector for https://github.com/dantreiman/neural-style-tf#transfer-style-but-not-color
  4. Configuration options for https://github.com/dantreiman/neural-style-tf#single-image-or-video-frames - these look quite complex and would likely need explanations from Dan, as the optical flow part is going to be the most difficult part of this whole thing.
mattorp commented 3 years ago

Nice, seems doable. The optical flow seems like a clever idea, and it will be interesting to see how it plays with the different types of scenes.