jfkirk / tensorrec

A TensorFlow recommendation algorithm and framework in Python.
Apache License 2.0
1.27k stars 223 forks source link

Thread safe #95

Open alimaan2935 opened 5 years ago

alimaan2935 commented 5 years ago

I am building a recommendation system with tensorrec on a tornado server. When i run the system sync, it works fine. But when i run this async tornado server, i get an error saying: ValueError: Fetch argument <tf.Operation 'make_initializer_3' type=MakeIterator> cannot be interpreted as a Tensor. (Operation name: "make_initializer_3" op: "MakeIterator" input: "TensorSliceDataset_3" input: "Iterator_3" attr { key: "_class" value { list { s: "loc:@Iterator_3" } } } is not an element of this graph.)

Is it the problem that the library is not thread safe? or am i missing something here?

jfkirk commented 5 years ago

Hey @alimaan2935 -- Thanks for reporting! Is there a full stack trace for that error? The lack of thread safety may be within TensorRec or within the TensorFlow ops themselves, so I'd like to isolate the culprit.

alimaan2935 commented 5 years ago

Thanks for quick reply. This is the the complete stack trace: `[I 180830 08:31:34 tensorrec:589] Processing interaction and feature data
[E 180830 08:31:34 base_events:1259] Future exception was never retrieved
future: <Future finished exception=ValueError('Fetch argument <tf.Operation \'make_initialize 3\' type=MakeIterator> cannot be interpreted as a Tensor. (Operation name: "make_initializer_3"\n : "MakeIterator"\ninput: "TensorSliceDataset_3"\ninput: "Iterator_3"\nattr {\n key: "_class"\n
lue {\n list {\n s: "loc:@Iterator_3"\n }\n }\n}\n is not an element of this graph.)'

Traceback (most recent call last):
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 282, in in
fetch, allow_tensor=True, allow_operation=True))
File "C:\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 3339, in as_g ph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 3423, in as aph_element_locked
raise ValueError("Operation %s is not an element of this graph." % obj)
ValueError: Operation name: "make_initializer_3"
op: "MakeIterator"
input: "TensorSliceDataset_3"
input: "Iterator_3"
attr {
key: "_class"
value {
list {
s: "loc:@Iterator_3"
}
}
}
is not an element of this graph.

During handling of the above exception, another exception occurred:                          

Traceback (most recent call last):                                                           
  File "C:\Python35\lib\site-packages\tornado\gen.py", line 1141, in run                     
    yielded = self.gen.throw(*exc_info)                                                      
  File "C:\Users\aliir\Desktop\WorkFiles\recommender-api\app\handler.py", line 55, in predict
    results = yield self._blocking_predict(data)                                             
  File "C:\Python35\lib\site-packages\tornado\gen.py", line 1133, in run                     
    value = future.result()                                                                  
  File "C:\Python35\lib\asyncio\futures.py", line 294, in result                             
    raise self._exception                                                                    
  File "C:\Python35\lib\concurrent\futures\thread.py", line 55, in run                       
    result = self.fn(*self.args, **self.kwargs)                                              
  File "C:\recommender-api\app\handler.py", line 49, in _blocki

_predict
results = get_real_time_recommendations(data, model)
File "C:\recommender-api\apis\recommendations.py", line 43, i get_real_time_recommendations
model.fit(user_data, properties_data, interactions, columns_metadata_dict)
File "C:\recommender-api\algos\prediction_model.py", line 85, n fit
epochs=self.epochs, learning_rate=self.learning_rate, n_sampled_items=self.n_sampled_item verbose=self.verbose)
File "C:\Python35\lib\site-packages\tensorrec\tensorrec.py", line 539, in fit
n_sampled_items=n_sampled_items)
File "C:\Python35\lib\site-packages\tensorrec\tensorrec.py", line 602, in fitpartial
, n_user_features = get_dimensions_from_tensorrec_dataset(first_batch[1])
File "C:\Python35\lib\site-packages\tensorrec\input_utils.py", line 67, in get_dimensions_f m_tensorrec_dataset
session.run(initializer)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 877, in run run_metadata_ptr)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1085, in _ru self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 427, in in
self._fetch_mapper = _FetchMapper.forfetch(fetches)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 253, in for
tch
return _ElementFetchMapper(fetches, contraction_fn)
File "C:\Python35\lib\site-packages\tensorflow\python\client\session.py", line 289, in in
'Tensor. (%s)' % (fetch, str(e)))
ValueError: Fetch argument <tf.Operation 'make_initializer_3' type=MakeIterator> cannot be in rpreted as a Tensor. (Operation name: "make_initializer_3"
op: "MakeIterator"
input: "TensorSliceDataset_3"
input: "Iterator_3"
attr {
key: "_class"
value {
list {
s: "loc:@Iterator_3"
}
}
}
is not an element of this graph.)
`