Version incompatibilities with function preprocessor

[rpasricha]

When packaging the code using Python 3.5 and running the code with a Python 3.6 base image, the interpreter crashes:

XXX lineno: 13, opcode: 0
/opt/conda/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
Traceback (most recent call last):
  File "/app/function_shim.py", line 15, in <module>
    call(args.serialized_fn_file)
  File "/app/function_shim.py", line 7, in call
    res = fn()
  File "main.py", line 13, in train
    hostname = tf.constant(os.environ['HOSTNAME'])
SystemError: unknown opcode

[carlthome]

Which is particularly confusing with the GCP deployer, as AI platform is still only on Python 3.5 AFAIK.

This crashes with the same error:

import fairing

GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
DOCKER_REGISTRY = 'gcr.io/{}/fairing-job'.format(GCP_PROJECT)
fairing.config.set_builder('append', base_image='tensorflow/tensorflow:latest-py3', registry=DOCKER_REGISTRY)
fairing.config.set_preprocessor('function')
fairing.config.set_deployer('gcp')

def train():
    print("Hello world!")

remote_train = fairing.config.fn(train)
remote_train()

And unfortunately there doesn't seem to be a GCP AI Platform compatible base image with the latest stable TensorFlow release in TensorFlow's own container registry:

Images built after May 20 2019 (TF nightly, plus TF versions 1.14 and onward) are based on Ubuntu 18.04.

(source)

[carlthome]

As a workaround, this image seems to work fine: gcr.io/deeplearning-platform-release/tf-gpu.1-14

[chrishmorris]

fairing 0.5.3 has requirement urllib3==1.24.2, which is unpleasantly exact.

[jtfogarty]

/area engprod /priority p2