yushenxiang
commented
6 years ago 2017-11-10 videoGAN 阅读总结： 功能：1）generate videos from scratch (not conditioned on the past） 2） generate a sequence of frames (32 frames) 训练输入： a large amount of unlabeled video, 噪声 训练输出：video 测试输入：噪声 测试输出：video 数据处理： 1）降低相机抖动影响We extract SIFT keypoints [22], use RANSAC to estimate a homography (rotation, translation, scale) between adjacent frames, and warp frames to minimize background motion. 2）The only other pre-processing we do is normalizing the videos to be in the range [−1, 1]. 3）We extract frames at native frame rate (25 fps). We use 32-frame videos of spatial resolution 64 × 64. 结构： G: 两个独立的部分，输入一个100维的高斯噪声作为隐变量，一个生成动态的前景，一个生成静态的背景，将两个通过一个mask进行相加得到video D: We design the architecture to be reverse of the foreground stream in the generator, replacing fractionally strided convolutions with strided convolutions (to down-sample instead of up-sample), and replacing the last layer to output a binary classification (real or not). 参数： 1）We use the Adam optimizer and a fixed learning rate of 0.0002 and momentum term of 0.5 2）The latent code has 100 dimensions, which we sample from a normal distribution 3） a batch size of 64. 4）We initialize all weights with zero mean Gaussian noise with standard deviation 0.01 成果与不足： the model usually learns to put motion on the right objects, one common failure mode is that the objects lack resolution. 模型结果评价方法： 1）We quantitatively evaluate our generation using a psychophysical two-alternative forced choice with workers on Amazon Mechanical Turk. We show a worker two random videos,and ask them “Which video is more realistic?” 2）baseline:训练一个autoencoder进行对比 We train an autoencoder over our data. The encoder is similar to the discriminator network (except producing 100 dimensional code), while the decoder follows the two-stream generator network
拓展应用：使用一张静态图生成video 结构改动： We utilize the same model as our two-stream model, however we must make one change in order to input the static image instead of the latent code. We can do this by attaching a five- layer convolutional network to the front of the generator which encodes the image into the latent space, similar to a conditional generative adversarial network. The rest of the generator and discriminator networks remain the same. loss函数改动： we add an additional loss term that minimizes the L1 distance between the input and the first frame of the generated image. 效果： 1）Although the extrapolations are rarely correct, they often have fairly plausible motions. 2）most common failure is that the generated video has a scene similar but not identical to the input image, such as by changing colors or dropping/hallucinating objects. 改进方向： the former could be solved by a color histogram normalization in post-processing ,the latter will require building more powerful generative models.
2017-11-10 1)Image-to-Image Translation with Conditional Adversarial Nets 阅读总结： 训练input: 一个图片和随机高斯噪声（dropout) ，训练output:一个逼真的和输入图片相关的图片 测试input: 一个图片和随机高斯噪声（dropout) 。 测试output:一个逼真的和输入图片相关的图片 G: U-NET是an encoder-decoder with skip connections* (encoder: C64-C128-C256-C512-C512-C512-C512-C512 U-Net decoder: CD512-CD1024-CD1024-C1024-C1024-C512-C256-C128 After the last layer in the decoder, a convolution is ap- plied to map to the number of output channels (3 in general, except in colorization, where it is 2), followed by a Tanh function. As an exception to the above notation, Batch- Norm is not applied to the first C64 layer in the encoder. All ReLUs in the encoder are leaky, with slope 0.2, while ReLUs in the decoder are not leaky.) D：论文采用70x70的patchGAN (C64-C128-C256-C512,After the last layer, a convolution is applied to map to a 1 dimensional output, followed by a Sigmoid function. As an exception to the above notation, BatchNorm is not applied to the first C64 layer. All ReLUs are leaky, with slope 0.2.) 优势：可以用于处理任意大的图片 loss函数：G⇤ = arg min max LcGAN (G, D) + lamdaLL1(G). （Adding both terms together (with lamda = 100) reduces these artifacts.） 训练细节： 1）Weights were initialized from a Gaussian distribution with mean 0 and standard deviation 0.02. 2） apply batch normalization，use batch size 1 for certain experiments and 4 for others，noting little difference