search

theislab / scgen

Single cell perturbation prediction
https://scgen.readthedocs.io
GNU General Public License v3.0
260 stars 52 forks source link

ValueError: Variable encoder/dense/kernel/Adam/ already exists if model is trained in a loop #11

Closed rene-rex closed 4 years ago

rene-rex commented 4 years ago

Hello

First of all congratulations for this impressive work! I am very curious to find out how far we can go with VAEs and single cell data.

Unfortunately, I encountered an error when I tried to train a model for different data sets in a loop. It seems the variable for the optimizer is not reset if a new instance of VAEArith is created. Here is short snippet to demonstrate the problem (tested against scgen 1.1.3 from pypi).

import scgen
import numpy as np
import pandas
from anndata import AnnData

for i in range(1, 3):

    data = pandas.DataFrame(np.random.random((i * 100, 50)))
    obs = pandas.DataFrame(index=data.index)
    obs['batch'] = (['BatchA'] * i * 50) + (['BatchB'] * i * 50)
    obs['cell_type'] = ['TypeA'] * i * 100
    adata = AnnData(data, obs)

    network = scgen.VAEArith(x_dimension=adata.shape[1],
                             model_path="./models/batch-" + str(i))
    network.train(train_data=adata, n_epochs=100)
    corrected_adata = scgen.batch_removal(network, adata)

And here is the traceback:

Traceback (most recent call last):
  File "loop-bug.py", line 17, in <module>
    network = scgen.VAEArith(x_dimension=adata.shape[1], model_path="./models/batch-" + str(i))
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/scgen/models/_vae.py", line 46, in __init__
    self._loss_function()
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/scgen/models/_vae.py", line 158, in _loss_function
    self.solver = tf.train.AdamOptimizer(learning_rate=self.learning_rate).minimize(self.vae_loss)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 413, in minimize
    name=name)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 597, in apply_gradients
    self._create_slots(var_list)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/adam.py", line 131, in _create_slots
    self._zeros_slot(v, "m", self._name)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 1155, in _zeros_slot
    new_slot_variable = slot_creator.create_zeros_slot(var, op_name)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 190, in create_zeros_slot
    colocate_with_primary=colocate_with_primary)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 164, in create_slot_with_initializer
    dtype)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 74, in _create_slot_var
    validate_shape=validate_shape)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable
    aggregation=aggregation)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable
    aggregation=aggregation)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable
    aggregation=aggregation)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter
    aggregation=aggregation)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 864, in _get_single_variable
    (err_msg, "".join(traceback.format_list(tb))))
ValueError: Variable encoder/dense/kernel/Adam/ already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/scgen/models/_vae.py", line 158, in _loss_function
    self.solver = tf.train.AdamOptimizer(learning_rate=self.learning_rate).minimize(self.vae_loss)
  File "/home/user/software/vens/scgen/lib64/python3.6/site-packages/scgen/models/_vae.py", line 46, in __init__
    self._loss_function()
  File "loop-bug.py", line 17, in <module>
    network = scgen.VAEArith(x_dimension=adata.shape[1], model_path="./models/batch-" + str(i))

This seems to be related to issue #10. However, I tried to test this code against the current master branch but the installation of scgen fails. I created a another issue (#12) for this.

rene-rex commented 4 years ago

Found a work-around: Add tf.reset_default_graph() before constructing a new model. This should probably go into the __init__ function.

Naghipourfar commented 4 years ago

Hey, Thanks a lot for the report. Fixed it. :)

Best, Mohsen