search

ajbrock / Neural-Photo-Editor

A simple interface for editing natural photos with generative neural networks.
MIT License
2.08k stars 199 forks source link

Theano optimization failed #16

Open domarps opened 6 years ago

domarps commented 6 years ago

Hi,

I am trying to reproduce the code on a V100 instance and I ran into the following issues when I ran python NPE.py

Do you have any recommendations on how we can reproduce your experimental setup in the form of a Dockerfile?

/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file "/home/ubuntu/.config/matplotlib/matplotlibrc", line #2
  (fname, cnt))
/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file "/home/ubuntu/.config/matplotlib/matplotlibrc", line #3
  (fname, cnt))
Loading weights
Compiling Theano Functions
ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617198864 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison

ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617838416 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison

ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617837136 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison

ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617737296 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison

ERROR (theano.gof.opt): Optimization failure due to: local_abstractconv_check
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2034, in process_node
    replacements = lopt.transform(node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/tensor/nnet/opt.py", line 500, in local_abstractconv_check
    node.op.__class__.__name__)
LocalMetaOptimizerSkipAssertionError: AbstractConv2d Theano optimization failed: there is no implementation available supporting the requested options. Did you exclude both "conv_dnn" and "conv_gemm" from the optimizer? If on GPU, is cuDNN available and does the GPU support it? If on CPU, do you have a BLAS library installed Theano can link against? On the CPU we do not support float16.

Traceback (most recent call last):
  File "NPE.py", line 19, in <module>
    model = IAN(config_path = 'IAN_simple.py', dnn = False)
  File "/home/ubuntu/Neural-Photo-Editor/API.py", line 51, in __init__
    self.Z_hat_fn = theano.function([self.X],self.Z_hat)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
    output_keys=output_keys)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1839, in orig_function
    name=name)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1519, in __init__
    optimizer_profile = optimizer(fgraph)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 108, in __call__
    return self.optimize(fgraph)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 97, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 251, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 97, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2143, in apply
    nb += self.process_node(fgraph, node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2039, in process_node
    lopt, node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 1933, in warn_inplace
    return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 1919, in warn
    raise exc
theano.gof.opt.LocalMetaOptimizerSkipAssertionError: AbstractConv2d Theano optimization failed: there is no implementation available supporting the requested options. Did you exclude both "conv_dnn" and "conv_gemm" from the optimizer? If on GPU, is cuDNN available and does the GPU support it? If on CPU, do you have a BLAS library installed Theano can link against? On the CPU we do not support float16.
ajbrock commented 6 years ago

I've not used Docker at all (well, I've not successfully used Docker at all), but I can try to point you in the right direction. I was running this with python 2.7.12, Theano 0.7 or 0.8, and Lasagne 0.2 back in 2016, on GTX 980s and my laptop GPU for inference (GT730m).

At the end of the day though, I'm not sure you'll be able to run it on the V100s, what with Theano being deprecated and this code being 2+ years old at this point.

If you just want to use the interface, it should be trivial to plug in a PyTorch/Chainer/TF pre-trained GAN, just implement the appropriate methods in API.py.