Integration with AWS SageMaker Random Cut Forest algorithm for anomaly detection

[tkamenov-expedia]

https://aws.amazon.com/blogs/machine-learning/use-the-built-in-amazon-sagemaker-random-cut-forest-algorithm-for-anomaly-detection/

This is largely R&D work at this point, aimed at proving that we have an integration strategy for AWS SageMaker. Benefits include:

• Having access to trained models, in addition to the simple stream filters we already have. In particular, in the future we intend to integrate our internally-developed Aquila model, which we train offline, and so SageMaker integration will give us an option here.
• Having access to new AWS-provided anomaly detection models as they appear.
• Having a way to interface with non-SageMaker ML pipelines. (Expedia Group has multiple such pipelines, and it would be great to be able to integrate with any of them.)
• Having access to models developed in ML frameworks such as TensorFlow, MXNet, etc.
• Lowering the bar for developing, training and deploying new anomaly detection models.

Some questions we want to resolve:

• How do we want to use the SageMaker models? Does AA call models running live in SageMaker? Does AA read the models from S3 and then apply them locally as filters? Both?
• Relatedly, what are the performance characteristics of the two approaches I just described?
• Also relatedly, which format(s) does SageMaker use when storing the model? Is there some Java SDK for reading those models into AA, or would we need to build Java versions along with a deserializer? We need to resolve this early on to ensure we have some reasonable way to read the models.
• What's the integration strategy for supporting multiple pipelines? For example, do we want pipelines to dump models in a common format to a common location (e.g. transform to PMML and store in known S3 location)? Do we implement a service provider interface with provider-specific plugins? Something else?

[williewheeler]

For using SageMaker models, we are looking at saving them as Docker images (AWS supports this) and then running them as sidecars. That way AA can make web service calls to the models locally. Thanks Tanja for this idea.

[williewheeler]

Done: 77eab4a5755f666bc9b460c652ba148c785249ee