search

chuanli11 / CNNMRF

code for paper "Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis"
MIT License
864 stars 148 forks source link

Accelerate for multiple image input? #21

Closed LC1332 closed 7 years ago

LC1332 commented 8 years ago

Hi Chuan,

Fantastic work! I find it on CVPR list and the result of code is really impressive.

I have a question, if i have large number of image to transfer it (say like video input) it will take a very long time.

Can I accelrate with batch input or something?

I'm a new guy of torch, is that possible you roughly figure out the way to help me to implement it?

jbgrant01 commented 8 years ago

Since Chuan (Chris) hasn't responded I will give it a shot. You can run a bash shell script to process a large number of files, but it does take a long time. If you are trying to process a video, the results may not work out well. You may want to take a look at the paper in [http://github.com/manuelruder/artistic-videos] for the reasons why. His setup is also based on Justin Johnson's neural-style and does not use Chris's CNNMRF code. It took almost two days on my paltry computer to render a 15 sec video clip.

I am attempting to incorporate CNNMRF into the artistic-videos setup, but I am a hobbyist and may not be up to the task. Both CNNMRF and artistic-videos are heavily editted from neural-style, so I may not come up with anything. I think CNNMRF produced more pleasing output than neural-style and I think applying it to video would be amazing.

I can provide a bash script for CNNMRF if you want to proceed. I have only used one for neural-style and I would need to test it out on CNNMRF.

ttoinou commented 8 years ago

I'm trying to do the same. Before thinking about optical flow one speed up would be to take out of the frame loop operations that are the same for every frame (if the style image is constant during the video).

jbgrant01 commented 8 years ago

I don't think I could make that work. The way I think artistic-videos works is that it computes the whole frame for content losses, but uses the optical flow as a mask for the previous frame so that it doesn't update; only the moving or most forward parts of the frame update.

There are two considerations why I am attempting to fork artistic-videos with CNNMRF code: one is that I think it will speed up the processing of video clips. Currently artistic-videos takes 300 - 800 iterations for the frames with movement before hitting the total relative loss setting. CNNMRF runs the image through 100 iterations of content loss and two iterations of the MRF processing. Even though the second run through the MRF takes a while, I think overall it will be a speed improvement over the neural-style processing.

The second consideration is that I have been getting better, clearer results with CNNMRF than I did with neural-style. I believe if I can create a marriage of artistic-videos and CNNMRF it will look even better than it does currently. That said, I am a hobbyist and not a great programmer. I have been able to do a mapping from neural-style to artistic-videos, but have not started on trying to map CNNMRF to either neural-style or artistic-videos. Perhaps I will get lucky. It may turn out to be easier to port the optical flow part from artistic-videos to CNNMRF than to port the MRF part of CNNMRF to artistic-videos.

chuanli11 commented 7 years ago

Hey, thanks a lot for the discussion. I close this issue because there have been plenty of recent progress in video stylization using feed-forward networks.

https://github.com/jcjohnson/fast-neural-style https://github.com/DmitryUlyanov/texture_nets https://github.com/chuanli11/MGANs