Open dgoldenberg-audiomack opened 3 years ago
Retrieving recs with respective ratings looks like this for me now:
# Get recommendations for user 42:
ratings_42, titles_42 = index(tf.constant(["42"]))
print(">> Recommendations for user 42:")
ratings_arr = ratings_42[0]
titles_arr = titles_42[0]
for idx in range(0, len(ratings_arr)):
print(f"\t{ratings_arr[idx]} -- {titles_arr[idx].numpy().decode('utf-8')}")
Toning down the prints to the console
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
logging.getLogger('tensorflow').setLevel(logging.FATAL)
also
model.fit(cached_train, epochs=3, verbose=0)
This doesn't remove all prints but it does remove quite a few.
k
parameter detailed in the docs.verbose
argument to Keras fit/evaluate
calls.Hi @maciejkula Maciej, thanks for your prompt and thorough responses. Could we have this ticket open for now, I think I'm going to have a couple more questions as I work through my POC. Thanks.
Hi @maciejkula Maciej
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
if True: # noqa: E402
import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_recommenders as tfrs
tf.get_logger().setLevel(logging.FATAL)
logging.getLogger("tensorflow").setLevel(logging.FATAL)
logging.getLogger("tensorflow_recommenders").setLevel(logging.FATAL)
@tf.autograph.experimental.do_not_convert
on functions to turn off Autograph prints/logging.1/5 [=====> ... ] - ETA: 0s - factorized_top_k/top_1_categorical_accuracy: 4.8828e-04
- factorized_top_k/top_5_categorical_accuracy: 0.0078
- factorized_top_k/top_10_categorical_accuracy: 0.0215
- factorized_top_k/top_50_categorical_accuracy: 0.1299
- factorized_top_k/top_100_categorical_accuracy: 0.2314
- loss: 32467.8496 - regularization_loss: 0.0000e+00
- total_loss: 32467.8496
Hyperparameter tuning is crucial.
'factorized_top_k/top_1_categorical_accuracy': 0.000699999975040555, 'factorized_top_k/top_5_categorical_accuracy': 0.009349999949336052, 'factorized_top_k/top_10_categorical_accuracy': 0.022549999877810478, 'factorized_top_k/top_50_categorical_accuracy': 0.12484999746084213, 'factorized_top_k/top_100_categorical_accuracy': 0.23270000517368317, 'loss': 28244.7734375, 'regularization_loss': 0, 'total_loss': 28244.7734375
Scaling/distributing.
Have a look at this tutorial for a starting point.
These look like nice strategies but they all seem targeted toward specific machines. What are some of the approaches if you're running on a 'utility' cluster, say, you're in AWS EMR. How does one integrate and scale TFRS there?
Any idea on this?
Trying the MirroredStrategy with the MovieLens/retrieval type of sample, on Spark (AWS EMR).
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
movies, test, train, unique_movie_titles, unique_user_ids = prepare_data(movies, ratings)
model = create_and_train_model(movies, test, train, unique_movie_titles, unique_user_ids)
generate_recommendations(model, movies)
where
def create_and_train_model(movies, test, train, unique_movie_titles, unique_user_ids):
embedding_dimension = 32
# The query tower
user_model = tf.keras.Sequential(
[
tf.keras.layers.experimental.preprocessing.StringLookup(vocabulary=unique_user_ids, mask_token=None),
# We add an additional embedding to account for unknown tokens.
tf.keras.layers.Embedding(len(unique_user_ids) + 1, embedding_dimension),
]
)
# The candidate tower
movie_model = tf.keras.Sequential(
[
tf.keras.layers.experimental.preprocessing.StringLookup(vocabulary=unique_movie_titles, mask_token=None),
tf.keras.layers.Embedding(len(unique_movie_titles) + 1, embedding_dimension),
]
)
# Metrics
metrics = tfrs.metrics.FactorizedTopK(candidates=movies.batch(128).map(movie_model))
# Loss
task = tfrs.tasks.Retrieval(metrics=metrics)
cached_train = train.shuffle(100_000).batch(8192).cache()
cached_test = test.batch(4096).cache()
model = MovielensModel(user_model, movie_model, task)
model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1))
# train the model
model.fit(cached_train, epochs=3, verbose=0)
eval_results = model.evaluate(cached_test, return_dict=True)
return model
Got the below error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation sequential/embedding/embedding_lookup/ReadVariableOp: Could not satisfy explicit device specification '' because the node {{colocation_node sequential/embedding/embedding_lookup/ReadVariableOp}} was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=2 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
GatherV2: GPU CPU
Cast: GPU CPU
Const: GPU CPU
ResourceSparseApplyAdagradV2: CPU
_Arg: GPU CPU
ReadVariableOp: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
sequential_embedding_embedding_lookup_readvariableop_resource (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
adagrad_adagrad_update_1_update_0_resourcesparseapplyadagradv2_accum (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
sequential/embedding/embedding_lookup/ReadVariableOp (ReadVariableOp)
sequential/embedding/embedding_lookup/axis (Const)
sequential/embedding/embedding_lookup (GatherV2)
gradient_tape/sequential/embedding/embedding_lookup/Shape (Const)
gradient_tape/sequential/embedding/embedding_lookup/Cast (Cast)
Adagrad/Adagrad/update_1/update_0/ResourceSparseApplyAdagradV2 (ResourceSparseApplyAdagradV2) /job:localhost/replica:0/task:0/device:GPU:0
[[{{node sequential/embedding/embedding_lookup/ReadVariableOp}}]] [Op:__inference_train_function_1206667]
@dgoldenberg-audiomack, it appears Adagrad is missing some GPU implementation aspects in TF, as pointed out in issue #138 Perhaps try SGD or Adam optimizer.
Thanks, @davidcereal! This definitely has fixed that issue for me. I've peeked through the issue list for TF but didn't see anything related to this particular thing (OK, I see 138 here, thanks)
Apologies, a noob here but I suspect my questions will be repeated by other noobs :)
Could the developers please elaborate on the below (I think these could be good additions to the tutorials too).
_, titles = index(tf.constant(["42"]))
print(f"Recommendations for user 42: {titles[0, :3]}")
This retrieves the movie titles, in the tutorial, but not the rating values. (OK, x, titles = index(tf.constant(["42"])) -- looks like the x tensor contains the ratings)Thanks.