SagemakerEstimator in Spark ML Pipeline issue

[hdamani09]

Reference: MLFW-2726

System Information

• Spark
• SDK Version:spark_2.2.0-1.2.5
• Spark Version:2.4.3
• Algorithm: XGBoost

Describe the problem

Hi, I tried to add a sagemakerEstimator within a Spark ML Pipeline and fit the training dataset on the pipeline which worked without any issues. When I tried to save the pipeline itself, it threw an exception stating the pipeline contains a stage that is not writable. Is it intended to be that way since when fit runs on the sagemakerEstimator, it automatically persists the model to trainingOutputS3DataPath ? If I wish to have a pipeline persisted which contains other transformer stages along with the sagemakerEstimator instance how would I do it?

[ChoiByungWook]

Reference: MLFW-2726

Hello @hdamani09,

Apologies for the late reply.

Thank you for bringing this to our attention. I have created an internal backlog ticket to track this, as it seems that our SageMaker estimators don't have an implemented write function, which enables saving.

Is it intended to be that way since when fit runs on the sagemakerEstimator, it automatically persists the model to trainingOutputS3DataPath ? If I wish to have a pipeline persisted which contains other transformer stages along with the sagemakerEstimator instance how would I do it?

There doesn't seem to be a possible way to do this. I'll make note of this to investigate and provide a solution in the internal ticket.

I apologize for the experience.