akhorlin
commented
3 years ago We are not experts in the SageMaker environment. In case of the Google Colab, there are couple of work-arounds: (1) You can use S3/GCS for storage of files/assets. For TF Hub, you can location where temporary files are stored using TFHUB_CACHE_DIR, (2) another options is described here
Issue:
I tried loading a saved Keras model which consists of hub.KerasLayer with universal-sentence-encoder-multilingual-large which was saved during SageMaker training job. There are no errors during training, the training code used is given below. Dockerfile used to create the instance is given below.
But I am unable to load it using
load_model("model.h5", custom_objects={"KerasLayer":hub.KerasLayer})when trying in Jupyter notebook (SageMaker notebook), as it is looking for a hard-coded OS path during loading. The error and prediction code used are given below.Error:
I created
/opt/ml/code/google-use-multi-large/and~/opt/ml/code/google-use-multi-large/locally in the notebook environment, but the same error occurs.Click to expand: Dockerfile
``` Dockerfile FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.3.0-gpu-py37-cu102-ubuntu18.04 RUN mkdir -p /opt/ml/code WORKDIR /opt/ml/code RUN mkdir google-use-multi RUN wget "https://storage.googleapis.com/tfhub-modules/google/universal-sentence-encoder-multilingual-large/3.tar.gz" && \ tar -zxvf 3.tar.gz --directory google-use-multi-large/ ENV HUB_PATH='/opt/ml/code/google-use-multi/' ```Click to expand: Training Code
``` python import tensorflow as tf import tensorflow_text import tensorflow_hub as hub module_url = os.environ["HUB_PATH"] def build_model(module_url, num_classes): inputs = Input(shape=(1,), dtype=tf.string) embed_hublayer = hub.KerasLayer( module_url, input_shape=[], dtype=tf.string, trainable=False, name="USE_embedding", ) embedding = embed_hublayer(tf.squeeze(tf.cast(inputs, tf.string))) x = Dense(256, activation="relu")(embedding) x = Dropout(0.3)(x) outputs = Dense(num_classes, activation="softmax")(x) model = Model(inputs=inputs, outputs=outputs) model.compile( loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"] ) logger.info(model.summary()) return model model = build_model(module_url=module_url, num_classes=num_classes) model.fit(x_train, y_train, batch_size=batch_size, epochs=args.epochs, callbacks=callbacks_list, validation_data=validation_data, shuffle=True, verbose=2) model.save("model.h5") ```Click to expand: Prediction Code
``` python import tensorflow as tf import tensorflow_text import tensorflow_hub as hub from tensorflow.keras.models import load_model model = load_model("model.h5", custom_objects={"KerasLayer":hub.KerasLayer}) ```Click to expand: Full Error Log
``` python --------------------------------------------------------------------------- OSError Traceback (most recent call last)Version Info:
Would like to know how can I overcome this error of hard-coded OS path and load the saved Keras model in a new environment other than where it was trained such as in a SageMaker notebook/Google Colab notebook or how can I modify
build_modelfunction in training script?