Cannot create model version with custom prediction routine with scikit-learn beta feature

[rachels-sq]

Describe the bug I'm trying to create a model version with the custom prediction routine beta feature. The create keeps failing with an Unexpected error. I've tried creating the version through iTerm and in the AI Platform UI.

What sample is this bug related to? https://cloud.google.com/ml-engine/docs/scikit/custom-prediction-routine-scikit-learn

Source code / logs Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: 0 (Error code: 0)"

Full traceback:

DEBUG: Running [gcloud.beta.ai-platform.versions.create] with arguments: [--model: "test_model", --origin: "gs://test_model/models/", --package-uris: "[u'gs://test_model/models/my_custom_code-0.1.tar.gz']", --prediction-class: "predictor.MyPredictor", --python-version: "3.5", --runtime-version: "1.13", --verbosity: "debug", VERSION: "version_test"]
Creating version (this might take a few minutes)......failed.
DEBUG: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error:  "Failed to load model: Unexpected error when loading the model: 9 (Error code: 0)"
Traceback (most recent call last):
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 985, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 795, in Run
    resources = command_instance.Run(args)
  File "/Users/user_1/google-cloud-sdk/lib/surface/ai_platform/versions/create.py", line 158, in Run
    package_uris=args.package_uris)
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/versions_util.py", line 113, in Create
    message='Creating version (this might take a few minutes)...')
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/versions_util.py", line 74, in WaitForOpMaybe
    return operations_client.WaitForOperation(op, message=message).response
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/api_lib/ml_engine/operations.py", line 114, in WaitForOperation
    sleep_ms=5000)
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 264, in WaitFor
    sleep_ms, _StatusUpdate)
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 326, in PollUntilDone
    sleep_ms=sleep_ms)
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 229, in RetryOnResult
    if not should_retry(result, state):
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
  File "/Users/user_1/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 122, in IsDone
    raise OperationError(operation.error.message)
OperationError: Create Version failed. Bad model detected with error:  "Failed to load model: Unexpected error when loading the model: 0 (Error code: 0)"
ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error:  "Failed to load model: Unexpected error when loading the model: 9 (Error code: 0)"

System Information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Mojave 10.14.4
• Framework and version (Tensorflow, scikit-learn, XGBoost): scikit-learn
• Python version: 3.5.0
• Exact command to reproduce:

gcloud beta ai-platform versions create $VERSION_NAME \
 --model $MODEL_NAME \
 --runtime-version 1.13 \
 --python-version 3.5 \
 --origin gs://$MODEL_NAME/models/ \
 --package-uris gs:/$MODEL_NAME/models/my_custom_code-0.1.tar.gz \
 --prediction-class predictor.MyPredictor

Additional context I've tried updating sklearn to version 0.20.2 after finding this issue with no success: https://github.com/GoogleCloudPlatform/cloudml-samples/issues/419

I suspect I might need to include a preprocessor in my custom routine as well (I only need a custom predictor so didn't include a preprocessor from the start).

[nnegrey]

Hi @alecglassford, mind taking a look? Thanks

[rachels-sq]

I was using the wrong version of joblib.