feature request: train model locally, upload to sagemaker

[tmastny]

@jcpsantiago

Thoughts I sent over email:

---

As for training locally, I haven’t tried it myself but it’s possible in theory! I already have functionality to download a trained model and low it into R. The reverse should also be possible.

You can see my downloading process here: https://github.com/tmastny/sagemaker/blob/4d835ef3f3fbfa64a2ac5dd616eeb491ed3a3f39/R/helpers.R#L157-L169

Basically, sagemaker models are saved in a standard location, you can find with this: https://gist.github.com/tmastny/fc7dc459da58bc9d9da1435df1aaabf2

The model is a standard Python Xgboost model. Then it’s compressed as a pickle, and then as tar.gz.

It should be a matter of reversing this processing.

There are two complications:

1. Sagemaker expects a Python Xgboost model, so you likely cannot use the standard library(xgboost). There are ways to convert from the R to the Python version, but it’s doesn’t always work since the R version of xgboost is usually a few versions behind.

2. You’d have to create the a dummy Sagemaker estimator that thinks it’s trained, and then actually put the model into the S3 path it expects. Shouldn't be too hard in theory again, but again never tested it.

[jcpsantiago]

Def doable for TF and MXNet https://aws.amazon.com/blogs/machine-learning/bring-your-own-pre-trained-mxnet-or-tensorflow-models-into-amazon-sagemaker/

Didn't find a tutorial for XGBoost. I'm spending some time on it though, independent of language.

[tmastny]

Great find! That gives me confidence the process I laid out should work. From your link, it looks like the model_data parameter is key when you are constructing the estimator with a pre-trained model.

Risk number 1. is still a problem. Do you intend to train Xgboost using the xgboost R package?

Here's some testing I did a while ago, and it didn't work great: https://gist.github.com/tmastny/c2d8a70324d8ff328559f7471e427119

However, the R package has been updated recently: https://cran.r-project.org/web/packages/xgboost/index.html Definitely should start testing again.

[tmastny]

For the generalized Estimator construct we use, it looks like model_uri could be the parameter we need:

https://sagemaker.readthedocs.io/en/latest/estimators.html#sagemaker.estimator.EstimatorBase

---

model_uri (str) – URI where a pre-trained model is stored, either locally or in S3 (default: None). If specified, the estimator will create a channel pointing to the model so the training job can download it. This model can be a ‘model.tar.gz’ from a previous training job, or other artifacts coming from a different source.

In local mode, this should point to the path in which the model is located and not the file itself, as local Docker containers will try to mount the URI as a volume.

More information: https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-training.html#td-deserialization

---

[tmastny]

Or maybe EstimatorBase.create_model: https://sagemaker.readthedocs.io/en/latest/estimators.html#sagemaker.estimator.EstimatorBase.create_model

Those arguments are sent to the sagemaker.model.Model constructor: https://sagemaker.readthedocs.io/en/latest/model.html#sagemaker.model.Model

There we do have the model_data parameter found in the example documentation for TF.

model_data (str) – The S3 location of a SageMaker model data .tar.gz file.

[jcpsantiago]

One way is to use your own docker container for inference, which allows you to use any algorithm you want https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html

I'm putting together a demo repo. {sagemaker} would then be used to upload the model to S3 and create the sagemaker endpoint, pointing at the model and a custom ECR docker image.