Open alimaan2935 opened 5 years ago
Hey @alimaan2935 -- Thanks for reporting! Is there a full stack trace for that error? The lack of thread safety may be within TensorRec or within the TensorFlow ops themselves, so I'd like to isolate the culprit.
Thanks for quick reply. This is the the complete stack trace:
`[I 180830 08:31:34 tensorrec:589] Processing interaction and feature data
[E 180830 08:31:34 base_events:1259] Future exception was never retrieved
future: <Future finished exception=ValueError('Fetch argument <tf.Operation \'make_initialize
3\' type=MakeIterator> cannot be interpreted as a Tensor. (Operation name: "make_initializer_3"\n
: "MakeIterator"\ninput: "TensorSliceDataset_3"\ninput: "Iterator_3"\nattr {\n key: "_class"\n
lue {\n list {\n s: "loc:@Iterator_3"\n }\n }\n}\n is not an element of this graph.)'
Traceback (most recent call last):
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 282, in in
fetch, allow_tensor=True, allow_operation=True))
File "C:\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 3339, in as_g ph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 3423, in as aph_element_locked
raise ValueError("Operation %s is not an element of this graph." % obj)
ValueError: Operation name: "make_initializer_3"
op: "MakeIterator"
input: "TensorSliceDataset_3"
input: "Iterator_3"
attr {
key: "_class"
value {
list {
s: "loc:@Iterator_3"
}
}
}
is not an element of this graph.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python35\lib\site-packages\tornado\gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "C:\Users\aliir\Desktop\WorkFiles\recommender-api\app\handler.py", line 55, in predict
results = yield self._blocking_predict(data)
File "C:\Python35\lib\site-packages\tornado\gen.py", line 1133, in run
value = future.result()
File "C:\Python35\lib\asyncio\futures.py", line 294, in result
raise self._exception
File "C:\Python35\lib\concurrent\futures\thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\recommender-api\app\handler.py", line 49, in _blocki
_predict
results = get_real_time_recommendations(data, model)
File "C:\recommender-api\apis\recommendations.py", line 43, i
get_real_time_recommendations
model.fit(user_data, properties_data, interactions, columns_metadata_dict)
File "C:\recommender-api\algos\prediction_model.py", line 85,
n fit
epochs=self.epochs, learning_rate=self.learning_rate, n_sampled_items=self.n_sampled_item
verbose=self.verbose)
File "C:\Python35\lib\site-packages\tensorrec\tensorrec.py", line 539, in fit
n_sampled_items=n_sampled_items)
File "C:\Python35\lib\site-packages\tensorrec\tensorrec.py", line 602, in fitpartial
, n_user_features = get_dimensions_from_tensorrec_dataset(first_batch[1])
File "C:\Python35\lib\site-packages\tensorrec\input_utils.py", line 67, in get_dimensions_f
m_tensorrec_dataset
session.run(initializer)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 877, in run
run_metadata_ptr)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1085, in _ru
self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 427, in in
self._fetch_mapper = _FetchMapper.forfetch(fetches)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 253, in for
tch
return _ElementFetchMapper(fetches, contraction_fn)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 289, in in
'Tensor. (%s)' % (fetch, str(e)))
ValueError: Fetch argument <tf.Operation 'make_initializer_3' type=MakeIterator> cannot be in
rpreted as a Tensor. (Operation name: "make_initializer_3"
op: "MakeIterator"
input: "TensorSliceDataset_3"
input: "Iterator_3"
attr {
key: "_class"
value {
list {
s: "loc:@Iterator_3"
}
}
}
is not an element of this graph.)
`
I am building a recommendation system with tensorrec on a tornado server. When i run the system sync, it works fine. But when i run this async tornado server, i get an error saying:
ValueError: Fetch argument <tf.Operation 'make_initializer_3' type=MakeIterator> cannot be interpreted as a Tensor. (Operation name: "make_initializer_3" op: "MakeIterator" input: "TensorSliceDataset_3" input: "Iterator_3" attr { key: "_class" value { list { s: "loc:@Iterator_3" } } } is not an element of this graph.)
Is it the problem that the library is not thread safe? or am i missing something here?