awslabs / sagemaker-debugger

Amazon SageMaker Debugger provides functionality to save tensors during training of machine learning jobs and analyze those tensors
Apache License 2.0
158 stars 82 forks source link

[Feature Request] TensorBoardOutputConfig local output path #560

Open austinmw opened 2 years ago

austinmw commented 2 years ago

Hi, I really wish I could use TensorBoardOutputConfig when my SageMaker Estimator instance type is local or local_gpu.

This could be accomplished by either or both of:

from sagemaker.debugger import TensorBoardOutputConfig

instance_type = 'local_gpu'

tb_output_path = "file:///local/path/to/stream/tensorboard/files/to"

tensorboard_output_config = TensorBoardOutputConfig(
   # and rename s3_output_path to output_path?
    output_path=tb_output_path,
    container_local_output_path='/tensorboard'
)

This would make it possible to perform local debugging of tensorboard logging code within my sagemaker scripts. Without local debugging, writing tensorboard logging code for sagemaker jobs can be a very slow process.