search

CompVis / adaptive-style-transfer

source code for the ECCV18 paper A Style-Aware Content Loss for Real-time HD Style Transfer
https://compvis.github.io/adaptive-style-transfer/
GNU General Public License v3.0
730 stars 140 forks source link

Training seems to collapse after 38,000 iterations #9

Closed alejodosr closed 5 years ago

alejodosr commented 5 years ago

Hi all,

First of all I want o congratulate you for your amazing work on style transfer :) I am currently training with your code in order to be able to generate semi-realistic images from simulated ones.

I am training with a Nvidia Tesla K80, and I see the following after certain iterations:

step_36500 Iteration 36500

step_37000 Iteration 37000

step_37500 Iteration 37500

step_38000 Iteration 38000

step_38500 Iteration 38500

Is this normal? Do you have any suggestion? It worth mention that I am filling the Places2 dataset with my own data (only in the folder /a/abbey). Thanks in advance.

dimakot55 commented 5 years ago

Well, that might be even the case for some models trained for style transfer task, it's a rare occasion still. Just try to continue training from the last checkpoint saved before model collapse. I shall add that we haven't tried to train the model for different datasets types like yours. Would be nice to hear some positive feedback.

asanakoy commented 5 years ago

closed due to inactivity