search

yahoo / TensorFlowOnSpark

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Apache License 2.0
3.88k stars 939 forks source link

Monitor Bandwidth Utilization of Nodes While Training #548

Open orwa-te opened 3 years ago

orwa-te commented 3 years ago

Environment:

Question: Is there a way to monitor the network utilization of nodes while communicating with each other to transfer the gradients in order to update the model? I want to measure the size of data sent from one node to another one for a single batch and all batches. I think that Tensorboard does not support such a feature

Spark Submit Command Line: spark-submit --master spark://master:7077 train_file.py --cluster_size 3 --epochs 10

leewyang commented 3 years ago

@orwa-te I'm not aware of anything within tensorflow which monitors cross-node traffic.

orwa-te commented 3 years ago

If I only need to know the exact data size of the buffer passed to AllReduce operation in Ring-AllReduce, where should I set a print log inside my code? More specifically, the amount(size) of data that is being sent from one node to another for the aggregation process after the gradients of one batch are computed?