dubois
commented
6 years ago This isn't supported out of the box by Tilt Brush Toolkit. Even in Tilt Brush (as you've seen), the offline rendering process for 360 videos doesn't support audio.
The best way forward that I can think of would involve using a third-party 360 renderer with the Toolkit. All the animation happens in the shaders; this is true both for time-based and audio-based animation. So instead of recording UnityEngine.Transform and interpolating them when rendering out a scene, you should be able to record the shader inputs and interpolate them (or easier: pick the closest value). So it would be something like:
- Play through your experience once, in real time, recording timestamps and the values of the shader inputs at that time
- Render out your experience, using the previously recorded data to set the shader inputs before recording each frame
The shader inputs are Unity's _Time.xyzw, and some textures and floatf4s ("_WaveFormTex", "_FFTTex", "_PeakBandLevels", etc) that you can find in Brush.cginc; search for "Common for Music Reactive Brushes".
Alternatively, instead of recording and manually setting up those music shader inputs, when rendering you could figure out the snip of audio corresponding to the current frame and send it into the audio analysis module -- that's the open source project Reaktion by Keijiro.
This is definitely an engineering-heavy project, but it seems doable. You may also want to post your question on the Tilt Brush product forums (https://productforums.google.com/forum/#!forum/tilt-brush) to see if any other users have done similar work.
So, these audio reactivity brushes are a real pain to capture in Unity. Tilt Brush documentations' technical way of downloading 360 videos doesn't allow for real-time sync with audio. By bringing Tilt Brush into Unity and I'm able to create a flawless app and custom interactions. But, try to capture everything in real-time 360 video from VR?!? Yeah right. All the programs, free and purchased, I've found cannot complete this task in real-time. The best only previews in real-time (OliVR). The programs are in alpha or beta on real-time 360 capture. I'm not even trying to move the camera through the painting. I just want one point through a 3 1/2 minute song.
Any ideas? Or will Tilt Brush/Unity allow for sync rendering with these brushes in some way I haven't discovered yet?
If you need more details about what I'm doing let me know. Normal people just animate objects, which are easy to capture. But these brushes are a new beast. Recording the animations in Unity doesn't seem to work, though most recorders just used transformation settings and the brushed don't touch those settings.
Susan