search

awslabs / sagemaker-debugger

Amazon SageMaker Debugger provides functionality to save tensors during training of machine learning jobs and analyze those tensors
Apache License 2.0
161 stars 83 forks source link

Compatibility with TF 2.7 #527

Closed adimux closed 2 years ago

adimux commented 2 years ago

...by fixing broken tests in test_keras_mirrored.py

Motive: Tensorflow 2.7.0 release

smdebug.tensorflow.get_hook() was returning None, which made many tests fail, but this was actually a false alarm. We just have to mark the version of tensorflow we are trying to release (2.7.0) as a supported framework version, so get_hook() doesn't return `None'...

I know what you think. It's not pythonic to silently fail and we should instead throw an exception, but this code preceded me and I don't want to introduce a breaking change (an exception being raised) and I don't want to introduce warnings either in case `get_hook()' gets called in unexpected ways multiple times during the lifecycle of a training job.

Instead, I just created a tests that alarms us of the situation when get_hook() might return None when it's actually just because we forgot to update a constant.

Description of changes:

Style and formatting:

I have run pre-commit install && pre-commit run --all-files to ensure that auto-formatting happens with every commit.

Issue number, if available

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.