search

awslabs / sagemaker-debugger

Amazon SageMaker Debugger provides functionality to save tensors during training of machine learning jobs and analyze those tensors
Apache License 2.0
161 stars 83 forks source link

Sagemaker Debugger with HPO #325

Open tvkpz opened 4 years ago

tvkpz commented 4 years ago

Can you please confirm if Sagemaker Debugger works with HPO. I get errors when the code that works perfectly fine with SM script mode fails when extended to HPO.

FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/config/debughookconfig.json'

Vikas-kum commented 4 years ago

Currently we dont have SageMaker Debugger support with HPO.

But if you wish to use smdebug and has training running in internet mode, you can create smdebug hook in training script and provide s3 path in out_dir like : s3://bucket_name/some_unique_prefix/ https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#hook-from-python-constructor