Scikit-Learn on a PowerFul VM vs Scikit Learn on the AI Platform

[gprinz]

Evaluate both, model training with large VMs (e.g. 100 cores) and the AI Platform.

AI Platform

• https://cloud.google.com/ml-engine/docs/training-overview
• https://cloud.google.com/ml-engine/docs/machine-types
• https://towardsdatascience.com/how-to-do-serverless-machine-learning-with-scikit-learn-on-google-cloud-ml-engine-db26dcc558a2

VMs

• https://cloud.google.com/compute/vm-instance-pricing

What are the differences between these two approaches? Which one is better in practice?

[tarrade]

Implementation done on using AI Platform notebook:

• price need to be check
• easier way to do hyperparameter tuning
• no parallel processing (limitation from scikit-learn)
• need to check how much data we can process

List of documentation: https://cloud.google.com/ml-engine/docs/scikit/training-scikit-learn?hl=en https://cloud.google.com/ml-engine/docs/training-jobs https://cloud.google.com/ml-engine/docs/training-at-scale?hl=en https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/notebooks/scikit-learn/Training%20with%20scikit-learn%20in%20CMLE.ipynb https://cloud.google.com/ml-engine/docs/training-overview https://cloud.google.com/ml-engine/docs/machine-types

https://cloud.google.com/ml-engine/docs/hyperparameter-tuning-overview https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs#parametertype https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs#HyperparameterSpec https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs#Algorithm https://github.com/GoogleCloudPlatform/cloudml-hypertune https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-on-google-cloud-platform-is-now-faster-and-smarter https://medium.com/@linda0511ny/automatic-hyperparameter-tuning-via-gcp-ml-engine-184a451e701e