Do video models dream of electrical sheeps in motion

[daniel-j-h]

Work in progress; let's see if this works out :hugs:

[daniel-j-h]

Here's where we are right now:

1. Create a random video with 32 frames of shape CxTxHxW 3x32x112x112
2. Hook this random tensor up in the computational tree so we can optimize it
3. Pass this random tensor through the trained video model up to the nth layer
4. Create a loss maximizing the the nth layer's activations by optimizing the input video

Here are example videos maximizing layer3 and layer4 activations:

Learnings

• Can use a very high learning rate e.g. 1e-2
• Should maximize activations for a single channel / volume, not mean over all
• Get high-frequency patterns fooling the network but not reasonable for humans

We then looked into regularization terms to get rid of the high-frequency patterns; the total variation regularization loss implemented right now results in very strong checkerboard patterns.

Next actions: figure out better regularization term and look into prior art how other folks handle this.

[daniel-j-h]

Completely different approach now; below are results for maximizing layer2 activations with different learning rates and number of iterations (second one is stronger) - I think this is the way to go :rocket:

More examples from: stem, layer1, layer2, layer3, layer4

     ---     

[sandhawalia]

Dayum 🔥

[daniel-j-h]

Here are a few examples when we - instead of optimizing all activations - select a specific channel; the activations are of shape NxCxTxHxW; below are a few examples for optimizing with fixed i in

acts[:,i, :, :, :].norm()

[daniel-j-h]

Starting with a random tensor instead of a seed video and optimizing all activations:

[daniel-j-h]

Starting from random tensor optimizing specific channels (3 and 6 in this case)

[daniel-j-h]

Bringing back the 3d total variation loss term and scaling it

seems to get rid of high frequencies in the output which makes it more pleasant to look at :hugs:

direct comparison:

• no total variation term
• (-1) 100 tv(video)
• (-1) 200 tv(video)

[daniel-j-h]

Changing how we normalize the gradients

[daniel-j-h]

Latest version maximizing all channels in layer2

Latest version maximizing channel 6 in layer2

[daniel-j-h]

Here are results for

• layer2 again
• maximizing specific TxHxW channel activations
• resizing to frame size of 248 px on the shorter edge

which barely fits on one of my GTX 1080 TIs:

Input clip from

youtube-dl -f 18 yJbXdOdTaJc

ffmpeg -i yJbXdOdTaJc.mp4 -ss 23:48 -to 23:53 -crf 23 -r 16 -an clips.mp4

[daniel-j-h]

Merging this into master as is right now. We can explore more advanced techniques such as dreaming at multiple scales and related in separate pull requests in the future.