search

Justin-Tan / generative-compression

TensorFlow Implementation of Generative Adversarial Networks for Extreme Learned Image Compression
MIT License
517 stars 105 forks source link

Killed when traning #4

Closed chenxianghu closed 6 years ago

chenxianghu commented 6 years ago

I am a new learner of tensorflow for only one week, this is the first time i use tensorflow, so there are many issues. During tensorflow installation and training data set preparation, i fix the issues i met. Now i will introduce how i run this demo:

  1. install tensorflow and the required libraries
  2. download Cityscapes leftImg8bit dataset and resampled to [512 x 1024] using imagemagick conert 3.train the model using cmd: python3 train.py but it seems the program is killed by Linux system because of high cpu & memory usage my Linux PC configuration is a little low Memory: 4G CPU: Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz 4 cores

do you have some suggestiones how can i run successfully on my Linux PC, is there some way to restrict the CPU & memory usage of tensorflow or reduce the traning image size or number, i hope your suggestiones, but anyway i should try my best to solve this issue, Thanks!

another thing, when will you upload your well-trained model and compress.py to github?

chenxh@chenxh-HP-Compaq-Pro-6380-MT:~/poc/ganforimagecompress/generative-compression$ python3 train.py Training on dataset cityscapes Building computational graph ... Training on cityscapes Training on cityscapes Real image shape: [None, None, None, 3] Reconstruction shape: [None, None, None, 3] Shape of x: [None, None, None, 3] Shape of x: [None, None, None, 3] Generator parameters: [<tf.Variable 'generator/encoder/conv2d/kernel:0' shape=(7, 7, 3, 60) dtype=float32_ref>, <tf.Var iable 'generator/encoder/conv2d/bias:0' shape=(60,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm/be ta:0' shape=(60,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm/gamma:0' shape=(60,) dtype=float32_r ef>, <tf.Variable 'generator/encoder/conv2d_1/kernel:0' shape=(3, 3, 60, 120) dtype=float32_ref>, <tf.Variable 'generat or/encoder/conv2d_1/bias:0' shape=(120,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_1/beta:0' sha pe=(120,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_1/gamma:0' shape=(120,) dtype=float32_ref>, <tf.Variable 'generator/encoder/conv2d_2/kernel:0' shape=(3, 3, 120, 240) dtype=float32_ref>, <tf.Variable 'generator/e ncoder/conv2d_2/bias:0' shape=(240,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_2/beta:0' shape=( 240,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_2/gamma:0' shape=(240,) dtype=float32_ref>, <tf. Variable 'generator/encoder/conv2d_3/kernel:0' shape=(3, 3, 240, 480) dtype=float32_ref>, <tf.Variable 'generator/encod er/conv2d_3/bias:0' shape=(480,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_3/beta:0' shape=(480, ) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_3/gamma:0' shape=(480,) dtype=float32_ref>, <tf.Vari able 'generator/encoder/conv2d_4/kernel:0' shape=(3, 3, 480, 960) dtype=float32_ref>, <tf.Variable 'generator/encoder/c onv2d_4/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_4/beta:0' shape=(960,) dt ype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_4/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/encoder/conv2d_5/kernel:0' shape=(3, 3, 960, 8) dtype=float32ref>, <tf.Variable 'generator/encoder/conv2d 5/bias:0' shape=(8,) dtype=float32_ref>, <tf.Variable 'generator/encoder/InstanceNorm_5/beta:0' shape=(8,) dtype=float3 2_ref>, <tf.Variable 'generator/encoder/InstanceNorm_5/gamma:0' shape=(8,) dtype=float32_ref>, <tf.Variable 'generator/ conv2d/kernel:0' shape=(3, 3, 8, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d/bias:0' shape=(960,) dtype=flo at32_ref>, <tf.Variable 'generator/InstanceNorm/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/Instan ceNorm/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_1/kernel:0' shape=(3, 3, 960, 960) dtyp e=float32_ref>, <tf.Variable 'generator/conv2d_1/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/Insta nceNorm_1/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_1/gamma:0' shape=(960,) dtype=f loat32_ref>, <tf.Variable 'generator/conv2d_2/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'genera tor/conv2d_2/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_2/beta:0' shape=(960,) dtype =float32_ref>, <tf.Variable 'generator/InstanceNorm_2/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator /conv2d_3/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_3/bias:0' shape=(960,) dt ype=float32_ref>, <tf.Variable 'generator/InstanceNorm_3/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generat or/InstanceNorm_3/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_4/kernel:0' shape=(3, 3, 960 , 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_4/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'gene rator/InstanceNorm_4/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_4/gamma:0' shape=(96 0,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_5/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Varia ble 'generator/conv2d_5/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_5/beta:0' shape=( 960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_5/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_6/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_6/bias:0' shap e=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_6/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variab le 'generator/InstanceNorm_6/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_7/kernel:0' shape =(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_7/bias:0' shape=(960,) dtype=float32_ref>, <tf.Var iable 'generator/InstanceNorm_7/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_7/gamma:0 ' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_8/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref> , <tf.Variable 'generator/conv2d_8/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_8/beta :0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_8/gamma:0' shape=(960,) dtype=float32_ref>, < tf.Variable 'generator/conv2d_9/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_9/b ias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_9/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_9/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_10/ker nel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d10/bias:0' shape=(960,) dtype=float32 ref>, <tf.Variable 'generator/InstanceNorm_10/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/Instance Norm_10/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_11/kernel:0' shape=(3, 3, 960, 960) dt ype=float32_ref>, <tf.Variable 'generator/conv2d_11/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/In stanceNorm_11/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_11/gamma:0' shape=(960,) dt ype=float32_ref>, <tf.Variable 'generator/conv2d_12/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable ' generator/conv2d_12/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_12/beta:0' shape=(960 ,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_12/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable ' generator/conv2d_13/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_13/bias:0' shap e=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_13/beta:0' shape=(960,) dtype=float32_ref>, <tf.Varia ble 'generator/InstanceNorm_13/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_14/kernel:0' sh ape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_14/bias:0' shape=(960,) dtype=float32_ref>, <tf .Variable 'generator/InstanceNorm_14/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_14/g amma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_15/kernel:0' shape=(3, 3, 960, 960) dtype=float 32_ref>, <tf.Variable 'generator/conv2d_15/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNor m_15/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_15/gamma:0' shape=(960,) dtype=float 32_ref>, <tf.Variable 'generator/conv2d_16/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator /conv2d_16/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_16/beta:0' shape=(960,) dtype= float32_ref>, <tf.Variable 'generator/InstanceNorm_16/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator /conv2d_17/kernel:0' shape=(3, 3, 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_17/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_17/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'gene rator/InstanceNorm_17/gamma:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_18/kernel:0' shape=(3, 3 , 960, 960) dtype=float32_ref>, <tf.Variable 'generator/conv2d_18/bias:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_18/beta:0' shape=(960,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_18/gamma:0' s hape=(960,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_transpose/kernel:0' shape=(3, 3, 480, 960) dtype=float32 _ref>, <tf.Variable 'generator/conv2d_transpose/bias:0' shape=(480,) dtype=float32_ref>, <tf.Variable 'generator/Instan ceNorm_19/beta:0' shape=(480,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_19/gamma:0' shape=(480,) dtype= float32_ref>, <tf.Variable 'generator/conv2d_transpose_1/kernel:0' shape=(3, 3, 240, 480) dtype=float32_ref>, <tf.Varia ble 'generator/conv2d_transpose_1/bias:0' shape=(240,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_20/beta :0' shape=(240,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_20/gamma:0' shape=(240,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_transpose_2/kernel:0' shape=(3, 3, 120, 240) dtype=float32_ref>, <tf.Variable 'generator /conv2d_transpose_2/bias:0' shape=(120,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_21/beta:0' shape=(120 ,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm_21/gamma:0' shape=(120,) dtype=float32_ref>, <tf.Variable ' generator/conv2d_transpose_3/kernel:0' shape=(3, 3, 60, 120) dtype=float32_ref>, <tf.Variable 'generator/conv2d_transpo se_3/bias:0' shape=(60,) dtype=float32_ref>, <tf.Variable 'generator/InstanceNorm22/beta:0' shape=(60,) dtype=float32 ref>, <tf.Variable 'generator/InstanceNorm_22/gamma:0' shape=(60,) dtype=float32_ref>, <tf.Variable 'generator/conv2d_1 9/kernel:0' shape=(7, 7, 60, 3) dtype=float32_ref>, <tf.Variable 'generator/conv2d19/bias:0' shape=(3,) dtype=float32 ref>] Discriminator parameters: [<tf.Variable 'discriminator/discriminator_original/conv2d/kernel:0' shape=(4, 4, 3, 64) dtyp e=float32_ref>, <tf.Variable 'discriminator/discriminator_original/conv2d/bias:0' shape=(64,) dtype=float32_ref>, <tf.V ariable 'discriminator/discriminator_original/conv2d_1/kernel:0' shape=(4, 4, 64, 128) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_original/conv2d_1/bias:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'discriminator/di scriminator_original/InstanceNorm/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_or iginal/InstanceNorm/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_original/conv2d _2/kernel:0' shape=(4, 4, 128, 256) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_original/conv2d_2/bia s:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_original/InstanceNorm_1/beta:0' shape=( 256,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_original/InstanceNorm_1/gamma:0' shape=(256,) dtype =float32_ref>, <tf.Variable 'discriminator/discriminator_original/conv2d_3/kernel:0' shape=(4, 4, 256, 512) dtype=float 32_ref>, <tf.Variable 'discriminator/discriminator_original/conv2d_3/bias:0' shape=(512,) dtype=float32_ref>, <tf.Varia ble 'discriminator/discriminator_original/InstanceNorm_2/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discrim inator/discriminator_original/InstanceNorm_2/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/disc riminator_original/conv2d_4/kernel:0' shape=(4, 4, 512, 1) dtype=float32_ref>, <tf.Variable 'discriminator/discriminato r_original/conv2d_4/bias:0' shape=(1,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_2/conv 2d/kernel:0' shape=(4, 4, 3, 64) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_2/conv2d/bia s:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_2/conv2d_1/kernel:0' shape=( 4, 4, 64, 128) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_2/conv2d_1/bias:0' shape=(128, ) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_2/InstanceNorm/beta:0' shape=(128,) dtype=f loat32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_2/InstanceNorm/gamma:0' shape=(128,) dtype=float32_r ef>, <tf.Variable 'discriminator/discriminator_downsampled_2/conv2d_2/kernel:0' shape=(4, 4, 128, 256) dtype=float32_re f>, <tf.Variable 'discriminator/discriminator_downsampled_2/conv2d_2/bias:0' shape=(256,) dtype=float32_ref>, <tf.Varia ble 'discriminator/discriminator_downsampled_2/InstanceNorm_1/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'di scriminator/discriminator_downsampled_2/InstanceNorm_1/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'discrimi nator/discriminator_downsampled_2/conv2d_3/kernel:0' shape=(4, 4, 256, 512) dtype=float32_ref>, <tf.Variable 'discrimin ator/discriminator_downsampled_2/conv2d_3/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/discrimi nator_downsampled_2/InstanceNorm_2/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_d ownsampled_2/InstanceNorm_2/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsam pled_2/conv2d_4/kernel:0' shape=(4, 4, 512, 1) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsample d_2/conv2d_4/bias:0' shape=(1,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/conv2d/kern el:0' shape=(4, 4, 3, 64) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/conv2d/bias:0' sh ape=(64,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/conv2d_1/kernel:0' shape=(4, 4, 6 4, 128) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/conv2d_1/bias:0' shape=(128,) dtype =float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled4/InstanceNorm/beta:0' shape=(128,) dtype=float32 ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/InstanceNorm/gamma:0' shape=(128,) dtype=float32_ref>, <t f.Variable 'discriminator/discriminator_downsampled_4/conv2d_2/kernel:0' shape=(4, 4, 128, 256) dtype=float32_ref>, <tf .Variable 'discriminator/discriminator_downsampled_4/conv2d_2/bias:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'di scriminator/discriminator_downsampled_4/InstanceNorm_1/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'discrimin ator/discriminator_downsampled_4/InstanceNorm_1/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'discriminator/d iscriminator_downsampled_4/conv2d_3/kernel:0' shape=(4, 4, 256, 512) dtype=float32_ref>, <tf.Variable 'discriminator/di scriminator_downsampled_4/conv2d_3/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_d ownsampled_4/InstanceNorm_2/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsamp led_4/InstanceNorm_2/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/ conv2d_4/kernel:0' shape=(4, 4, 512, 1) dtype=float32_ref>, <tf.Variable 'discriminator/discriminator_downsampled_4/con v2d_4/bias:0' shape=(1,) dtype=float32_ref>] Epoch 0 | Generator Loss: 70.112 | Discriminator Loss: 9.131 | Rate: 0 examples/s (118.63 s) [*] Killed

Justin-Tan commented 6 years ago

The architecture defined in the paper for the encoder/decoder is quite large, so may struggle to fit in a machine with less memory. I would try removing some of the residual blocks (maybe trying reducing the number from 9 -> 6) in the decoder function in network.py. Perhaps set multiscale=False and use_feature_matching_loss=False in config.py to disable some extra computations. This might reduce the quality of the reconstructed image, but I haven't tested this in detail. Instead of using the cityscapes dataset you could try 256x256 Imagenet images instead, the reduced image size should reduce the size of the kernels that will be defined during training.

chenxianghu commented 6 years ago

I find a more higher configuration PC(8 cores and 32G memory) to train the model, and I do the following modifications:

  1. resize the Cityscapes leftImg8bit dataset to [256x 256]
  2. num_epochs in config.py from 512->120 After 120 epochs training, i check the PDF files under generative-compression/samples/cityscapes, the gen file is more and more looks like the real file. So i think my training is effective, and I will continue to train the saved model. My question is: Can you upload the compress.py to github for a complete project? Now i am a new learner of tensorflow, I want to use compress.py to verify effect of the the model i trained. Thanks very much!
Justin-Tan commented 6 years ago

You can now compress images using compress.py.

chenxianghu commented 6 years ago

Thanks for your effort!