search

awslabs / sagemaker-debugger

Amazon SageMaker Debugger provides functionality to save tensors during training of machine learning jobs and analyze those tensors
Apache License 2.0
161 stars 83 forks source link

Update the Hook callback to be compatible with xgboost>1.3.0 callback style #616

Closed haixiw closed 1 year ago

haixiw commented 2 years ago

from xgboost version 1.3.0 and later, all the callbacks taken by xgb.train() must be instances of xgb.callback.TrainingCallback https://xgboost.readthedocs.io/en/stable/python/python_api.html#callback-api From the usage : https://github.com/awslabs/sagemaker-debugger/blob/master/smdebug/xgboost/hook.py#L64 If we pass Hook together with other callbacks which are instances of xgb.callback.TrainingCallback, an Assert Error will happen because of the check here : https://github.com/dmlc/xgboost/blob/v1.5.2/python-package/xgboost/training.py#L62

mabunday commented 2 years ago

Reference: https://github.com/aws/sagemaker-xgboost-container/pull/301

yl-to commented 1 year ago

Issue should be fixed by PR:https://github.com/awslabs/sagemaker-debugger/pull/648 Closing.