Describe the current behavior:
I can install and run the TPU profiler, but I can't get it to run in the background to profile the execution of a model.
1) I've tried running a python subprocess, like the example from this StackOverflow question, but it runs once before the model runs, and then stops.
2) I tried to add it to the Tensorboard, but I had the same issue. The TPU profiler is initiated, but it's not updated in the background. I've created a Google Storage Bucket with the required access and seen that it can read and write.
Describe the expected behavior:
If possible, I'd love a simple example on how to use the TPU profiler with either a subprocess, a magic command, or integrated into Tensorboad. Also, it would be great to not require a google storage bucket.
bhack
commented
2 years ago
Describe the current behavior: I can install and run the TPU profiler, but I can't get it to run in the background to profile the execution of a model.
!pip install --upgrade "cloud-tpu-profiler>=1.15.0rc1"!capture_tpu_profile --tpu=$TPU_NAME --monitoring_level=21) I've tried running a python subprocess, like the example from this StackOverflow question, but it runs once before the model runs, and then stops. 2) I tried to add it to the Tensorboard, but I had the same issue. The TPU profiler is initiated, but it's not updated in the background. I've created a Google Storage Bucket with the required access and seen that it can read and write.
Describe the expected behavior: If possible, I'd love a simple example on how to use the TPU profiler with either a subprocess, a magic command, or integrated into Tensorboad. Also, it would be great to not require a google storage bucket.