ValueError: Tensor(...) must be from same graph as Tensor(...)

[chstarr]

I am using talos to do hyperparameter optimization on a keras CNN that I am using to analyze 1D data. I am running into the error described in the title, and can't figure out how to proceed. Below, I have pasted the code that I think might be relevant to the issue. Please let me know if you can spot the issue. Thank you!

Here is the model definition: def cnnMP(x_train, y_train, x_val, y_val, params):

initialize the model

model = Sequential()

# define the first set of CONV => ACTIVATION => POOL layers
model.add(Conv1D(params['filters1'], params['kernelSize1'], padding="same", input_shape=(params['nBins'],1)))
model.add(Activation(params['activation']))
model.add(MaxPooling1D(pool_size=params['poolSize1'], strides=None))

# define the second set of CONV => ACTIVATION => POOL layers
model.add(Conv1D(params['filters2'], params['kernelSize2'], padding="same"))
model.add(Activation(params['activation']))
model.add(MaxPooling1D(pool_size=params['poolSize2'], strides=None))

# define the first FC => ACTIVATION layers
model.add(Flatten())
model.add(Dense(params['dimFC1']))
model.add(Activation(params['activation']))

# define the second FC layer
model.add(Dense(params['nClasses']))

# lastly, define the soft-max classifier
model.add(Activation("softmax"))

model.compile(loss=params['loss'], optimizer=params['opt'], metrics=["accuracy"])    

# train model 
out = model.fit(x_train, y_train, batch_size=params['batchSize'], epochs=params['epochs'], verbose=params['verbose'],
                validation_data=[x_val,y_val])

# return the constructed network architecture
return out, model

Here is where I set up the hyperparameter scan (just trying to look at batch size for now): p = { 'nBins': [512], 'nClasses': [2], 'batchSize': [32, 64, 128], 'epochs': [50], 'activation': ['relu'], 'filters1': [50], 'kernelSize1': [5], 'poolSize1': [2], 'filters2': [20], 'kernelSize2': [5], 'poolSize2': [2], 'dimFC1': [500], 'loss': ["categorical_crossentropy"], 'opt': [SGD(lr=0.01)], 'verbose': [0] }

t = ta.Scan(np.expand_dims(trainHist2,axis=2), trainLabels, params=p, model=mods.cnnMP)

I needed to use the np.expand(...) function to get my data to pass through keras. By the way, I did confirm that this model works well with keras on its own. The one difference is that I have to remove the [ ] brackets from the parameter list when passing into model.fit() because this function requires actual parameters, not iterable lists.

Finally, here is the error I get when running ta.Scan(...):

0%| | 0/3 [00:00<?, ?it/s]

33%|███▎ | 1/3 [00:03<00:07, 3.50s/it]Traceback (most recent call last):

File "", line 1, in runfile('C:/Users/cstarr/Documents/GitHub/MP_nnDetector/cnnMP_optimHP.py', wdir='C:/Users/cstarr/Documents/GitHub/MP_nnDetector')

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 786, in runfile execfile(filename, namespace)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/cstarr/Documents/GitHub/MP_nnDetector/cnnMP_optimHP.py", line 122, in t = ta.Scan(np.expand_dims(trainHist2,axis=2), trainLabels, params=p, model=mods.cnnMP)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\talos\scan\Scan.py", line 170, in init self._null = self.runtime()

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\talos\scan\Scan.py", line 175, in runtime self = scan_run(self)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\talos\scan\scan_run.py", line 18, in scan_run self = scan_round(self)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\talos\scan\scan_round.py", line 32, in scan_round _hr_out, self.keras_model = ingest_model(self)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\talos\model\ingest_model.py", line 10, in ingest_model self.round_params)

File "C:\Users\cstarr\Documents\GitHub\MP_nnDetector\modelDefs.py", line 38, in cnnMP validation_data=[x_val,y_val])

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\training.py", line 1010, in fit self._make_train_function()

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\training.py", line 509, in _make_train_function loss=self.total_loss)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper return func(*args, **kwargs)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\optimizers.py", line 196, in get_updates v = self.momentum m - lr g # velocity

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 935, in _run_op return tensor_oper(a.value(), *args, **kwargs)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 810, in binary_op_wrapper with ops.name_scope(None, op_name, [x, y]) as name:

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 6083, in enter g = _get_graph_from_inputs(self._values)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 5713, in _get_graph_from_inputs _assert_same_graph(original_graph_element, graph_element)

File "C:\Users\cstarr\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 5649, in _assert_same_graph original_item))

ValueError: Tensor("training/SGD/Variable:0", shape=(5, 1, 50), dtype=float32_ref) must be from the same graph as Tensor("SGD_1/momentum/read:0", shape=(), dtype=float32).

[mikkokotila]

I think you have to set talos.Scan( ... reset_tf_graph=False ... ).

Note that this is reset_graph=False from v.0.6 onwards.

Let me know if that works.

[chstarr]

Unforunately didn't work...here is what I saw:

t = ta.Scan(np.expand_dims(trainHists2,axis=2), trainLabels, params=p, model=mods.cnnMP, reset_tf_graph=False)

Traceback (most recent call last):

File "", line 1, in t = ta.Scan(np.expand_dims(trainHists2,axis=2), trainLabels, params=p, model=mods.cnnMP, reset_tf_graph=False)

TypeError: init() got an unexpected keyword argument 'reset_tf_graph'

[mikkokotila]

Try clear_tf_session instead. Sorry for the confusion.

[mikkokotila]

Closing this as resolved. Feel free to open a new issue if anything.