search

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
https://bentoml.com
Apache License 2.0
7.16k stars 792 forks source link

Facing "Failed precondition: Attempting to use uninitialized value ..... " error when using bento_svc.predict, and the layer indicated in the error keeps changing when attempting multiple times. #677

Closed jesudasvf closed 4 years ago

jesudasvf commented 4 years ago

Describe the bug After creating and saving the Speech emotion classification service with bentoml, tried to use the predict function of the model using bentoml_svc.predict() function. However I met with an "Failed precondition: Attempting to use uninitialized value conv2d_8/kernel".

To Reproduce Steps to reproduce the behavior: PART 1 1.Create the service SpeechEmoClassificationService

  1. Pack the model, Save the model
  2. Call the predict function of the model using svc created with bentoml
  3. Faced the "Failed precondition: Attempting to use uninitialized value conv2d_8/kernel" error.

PART 2

  1. Repeat All the steps from PART 1
  2. Faced the "Failed precondition: Attempting to use uninitialized value conv2d_2/kernel"

PART 3

  1. Repeat All the steps from PART 1
  2. Faced the "Failed precondition: Attempting to use uninitialized value conv2d_6/kernel"

PART4

  1. Get a prediction from the model using Keras predict function without bentoml
  2. It produced the predictions from the model

Expected behavior

  1. The bentoml svc should have returned the predictions from the model.
  2. The error should be consistent, however the layer mentioned in the error kept on changing on each of the three attempts made.

Screenshots/Logs

  1. Attached is a txt file with the console output with logging enabled for all the 3 attempts and the normal prediction using Keras only.

bentoml_svc_predict_issue.txt

  1. Code file for service creation, packing, saving and prediction SpeechEmoClassificationService.txt

Also attached in the SpeechEmoClassificationService.py file

Environment:

Additional context

  1. All workarounds to ensure that the problem is not with the Keras model was done to ensure that the errors are not due to the model .
  2. The model was tested with its stand-alone inference with Keras and tensorflow without bentoml.
parano commented 4 years ago

hi @jesudasvf, could you provide more details on how SpeechEmoClassificationService is defined? and are you using the KerasModelArtifact or the TensorflowSavedModelArtifact?

jesudasvf commented 4 years ago

hi @jesudasvf, could you provide more details on how SpeechEmoClassificationService is defined? and are you using the KerasModelArtifact or the TensorflowSavedModelArtifact?

hi @parano , the attached file Sp eechEmoClassificationService.txt has the code for definition of the SpeechEmoClassificationService. I'm including the code here for your quick reference. I'm using KerasModelArtifact in the service definition. 

import pandas as pd
import numpy as np
import keras
# from tensorflow.keras.preprocessing import sequence, text
import bentoml
from bentoml import api, env, BentoService, artifacts, ver, save, load
from bentoml.artifact import KerasModelArtifact
from bentoml.handlers import JsonHandler
import logging
import librosa
from sklearn.preprocessing import LabelEncoder
# import bentoml
from keras.models import model_from_json

bentoml.config().set('core', 'debug', 'true')
bentoml.utils.log.configure_logging(logging.DEBUG)

@ver(major=1, minor=4)
@artifacts([KerasModelArtifact('model')])
@env(auto_pip_dependencies=True, pip_dependencies=['tensorflow-gpu==1.14.0', 'numpy', 'librosa', 'sklearn'])
class SpeechEmoClassificationService(BentoService):

  def get_mfcc_feature(self, input_wav_file, n_mfccc, len_mfcc):
    # twav_data, tindex = librosa.effects.trim(wav_data, top_db=60)
    wav_data, wav_sr = librosa.load(str(input_wav_file), res_type='kaiser_fast',
                                    sr=22050 * 2)
    sr = np.array(wav_sr)
    twav_data, tindex = librosa.effects.trim(wav_data, top_db=60)
    data_mfccs = librosa.feature.mfcc(y=twav_data, sr=sr, n_mfcc=n_mfccc)
    while len(data_mfccs[0]) < len_mfcc:
      gap = len_mfcc - len(data_mfccs[0]) if (len_mfcc - len(data_mfccs[0]) < len(data_mfccs[0])) else len(
        data_mfccs[0])
      mfccs_2_rpt = data_mfccs[:, (len(data_mfccs[0]) - gap):len(data_mfccs[0])]
      data_mfccs = np.concatenate((data_mfccs, mfccs_2_rpt), axis=1)
    data_mfccs = data_mfccs[:, :len_mfcc]

    mfccthreedim = np.expand_dims(data_mfccs, axis=2)
    mfccthreedim = np.expand_dims(mfccthreedim, axis=0)

    return mfccthreedim

  @api(JsonHandler)
  def predict(self, parsed_json):
    input_wav_file = parsed_json['file']

    # load the audio file
    # generate the audio features
    len_mfcc = 256
    n_mfcc = 40
    features = self.get_mfcc_feature(input_wav_file, n_mfcc, len_mfcc)
    predictions = self.artifacts.model.predict_classes(features, batch_size=16)
    # pred_max_conf = predictions.argmax(axis=1)
    # pred_max_conf = pred_max_conf.astype(int).flatten()
    # feature_LE = LabelEncoder()
    # feature_LE.fit_transform(self.gender_emo_lbls)
    # predicted_feature = (feature_LE.inverse_transform((pred_max_conf)))
    return predictions

model_genderemoreco_filepath   = "../saved_models/gender_emotion/SpeechAllGenderEmotionRecognition_2D_M_202005.h5"
model_genderemoreco_json       = "../saved_models/gender_emotion/SpeechAllGenderEmotionRecognition_2D_M_202005.json"

def load_classifier(model_file, model_json_file):
  print("Loading the classifier model : ", model_file, "  ....")

  json_file = open(model_json_file)
  classifer_model_json = json_file.read()
  json_file.close()
  loaded_classifier = model_from_json(classifer_model_json)
  # load weights into the model
  loaded_classifier.load_weights(model_file)
  print(model_file, "   classifier model loaded !")
  return loaded_classifier

trained_model = load_classifier(model_genderemoreco_filepath, model_genderemoreco_json)

# 2) `pack` it with required artifacts
bento_svc = SpeechEmoClassificationService()
bento_svc.pack('model', trained_model)
# bento_svc.name = "SpeechEmoClassificationService"

# 3) save your BentoSerivce
saved_path = bento_svc.save()
print(saved_path)
print(bento_svc.predict({'file' : 'C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/test_wav_samples/RECORDED/kamal_m_ang.wav'}))

loaded_bento_svc = load(saved_path)

pred_val = loaded_bento_svc.predict({'file': 'C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/test_wav_samples/RECORDED/kamal_m_ang.wav'})

print(pred_val)
jesudasvf commented 4 years ago

hi @jesudasvf, could you provide more details on how SpeechEmoClassificationService is defined? and are you using the KerasModelArtifact or the TensorflowSavedModelArtifact?

hi @parano , the model is saved as Keras model h5 file, hence using KerasModelArtifact

parano commented 4 years ago

Hi @jesudasvf, could you try moving the class definition of SpeechEmoClassificationService to a separate python file and import it in the script that does model training and save? similar to how the quick start guide is organized?

It is recommended to always put the source code of your BentoService class into an individual Python file and check it into source control(e.g. git) along with your model training code. https://docs.bentoml.org/en/latest/concepts.html#creating-bentoservice

You can also try put if __name__ == "__main__": around your model loading and packing code, e.g.:

if __name__ == "__main__":
  trained_model = load_classifier(model_genderemoreco_filepath, model_genderemoreco_json)

  # 2) `pack` it with required artifacts
  bento_svc = SpeechEmoClassificationService()
  bento_svc.pack('model', trained_model)
  # bento_svc.name = "SpeechEmoClassificationService"

  # 3) save your BentoSerivce
  saved_path = bento_svc.save()
  print(saved_path)
  print(bento_svc.predict({'file' : 'C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/test_wav_samples/RECORDED/kamal_m_ang.wav'}))

  loaded_bento_svc = load(saved_path)

  pred_val = loaded_bento_svc.predict({'file': 'C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/test_wav_samples/RECORDED/kamal_m_ang.wav'})

  print(pred_val)
jesudasvf commented 4 years ago

hi @parano , thanks for the update on the issue.

I have modified the code files as requested and have attached them for your reference and review. a. Updated the SpeechEmoClassificationService.py to remove the service pack and prediction code. b. created a new py file spemoclass_service_pack.py where the Service is imported, packed and then prediction method is invoked.

SpeechEmoClassificationService.pdf spemoclass_service_pack.pdf

With this change, the output for the two attempts are given below. The "Failed precondition: attempting to use uninitialized value …" still exists and the layer indicated in each attempt is different. please refer the console output given below.

======================================== 1st Attempt

bentoml_svc_predict_issue_with_updated_Service_codefiles.txt

../saved_models/gender_emotion/SpeechAllGenderEmotionRecognition_2D_M_202005.h5 classifier model loaded ! WARNING:tensorflow:From C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\bentoml\artifact\keras_model_artifact.py:134: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-05-20 18:59:08,593] DEBUG - Using BentoML with local Yatai server [2020-05-20 18:59:08,802] DEBUG - Upgrading tables to the latest revision [2020-05-20 18:59:08,807] DEBUG - Created temporary directory: C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-fv0aybb7 [2020-05-20 18:59:11,568] DEBUG - copy_used_py_modules target_module_name: inference.SpeechEmoClassificationService, target_module_file: C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py [2020-05-20 18:59:11,568] DEBUG - Searching for dependant modules of inference.SpeechEmoClassificationService:C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py [2020-05-20 18:59:47,434] DEBUG - Ignoring deps within local site-packages path: ['C:\SL_Projects\SpeechEmotionRecognition\spemoenv\Scripts\python36.zip', 'C:\Python368\lib', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\win32', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\win32\lib', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\Pythonwin', 'C:\Python368\lib\site-packages', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv', 'C:\Python368'] [2020-05-20 18:59:47,434] DEBUG - User local module deps found: main [2020-05-20 18:59:47,440] DEBUG - Copying user local python dependencies: {'inference.SpeechEmoClassificationService': <module 'inference.SpeechEmoClassificationService' from 'C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py'>} [2020-05-20 18:59:47,441] DEBUG - Copying local python module 'C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py' [2020-05-20 18:59:47,442] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-fv0aybb7\SpeechEmoClassificationService' [2020-05-20 18:59:47,443] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-fv0aybb7\SpeechEmoClassificationService\artifacts' [2020-05-20 18:59:47,444] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-fv0aybb7\SpeechEmoClassificationService\inference' [2020-05-20 18:59:47,444] DEBUG - Done copying local python dependant modules [2020-05-20 18:59:47,656] INFO - BentoService bundle 'SpeechEmoClassificationService:1.4.20200520185908_90DA1E' saved to: C:\Users\Jesudas/bentoml/repository/SpeechEmoClassificationService\1.4.20200520185908_90DA1E [2020-05-20 18:59:47,657] DEBUG - BentoML in debug mode, keeping temp directory "C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-fv0aybb7" C:\Users\Jesudas/bentoml/repository/SpeechEmoClassificationService\1.4.20200520185908_90DA1E Traceback (most recent call last): File "C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/inference/spemoclass_service_pack.py", line 38, in print(bento_svc.predict({'file' : 'C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/test_wav_samples/RECORDED/kamal_m_ang.wav'})) File "C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py", line 52, in predict predictions = self.artifacts.model.predict_classes(features, batch_size=16) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\bentoml\artifact\keras_model_artifact.py", line 278, in predict_classes return self.keras_model.predict_classes(*args, *kwargs) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\engine\sequential.py", line 267, in predict_classes proba = self.predict(x, batch_size=batch_size, verbose=verbose) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\engine\training.py", line 1169, in predict steps=steps) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\engine\training_arrays.py", line 294, in predict_loop batch_outs = f(ins_batch) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in call return self._call(inputs) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(array_vals) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\tensorflow\python\client\session.py", line 1458, in call run_metadata_ptr) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value conv2d_1/bias [[{{node conv2d_1/bias/read}}]]

Process finished with exit code 1

==================================================== 2nd Attempt

../saved_models/gender_emotion/SpeechAllGenderEmotionRecognition_2D_M_202005.h5 classifier model loaded ! [2020-05-20 19:02:22,820] DEBUG - Using BentoML with local Yatai server [2020-05-20 19:02:22,979] DEBUG - Upgrading tables to the latest revision [2020-05-20 19:02:22,983] DEBUG - Created temporary directory: C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-xfoo3l_o [2020-05-20 19:02:25,469] DEBUG - copy_used_py_modules target_module_name: inference.SpeechEmoClassificationService, target_module_file: C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py [2020-05-20 19:02:25,469] DEBUG - Searching for dependant modules of inference.SpeechEmoClassificationService:C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py [2020-05-20 19:02:38,391] DEBUG - Ignoring deps within local site-packages path: ['C:\SL_Projects\SpeechEmotionRecognition\spemoenv\Scripts\python36.zip', 'C:\Python368\lib', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\win32', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\win32\lib', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\Pythonwin', 'C:\Python368\lib\site-packages', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv', 'C:\Python368'] [2020-05-20 19:02:38,391] DEBUG - User local module deps found: main [2020-05-20 19:02:38,396] DEBUG - Copying user local python dependencies: {'inference.SpeechEmoClassificationService': <module 'inference.SpeechEmoClassificationService' from 'C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py'>} [2020-05-20 19:02:38,397] DEBUG - Copying local python module 'C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py' [2020-05-20 19:02:38,398] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-xfoo3l_o\SpeechEmoClassificationService' [2020-05-20 19:02:38,399] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-xfoo3l_o\SpeechEmoClassificationService\artifacts' [2020-05-20 19:02:38,399] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-xfoo3l_o\SpeechEmoClassificationService\inference' [2020-05-20 19:02:38,400] DEBUG - Done copying local python dependant modules [2020-05-20 19:02:38,582] INFO - BentoService bundle 'SpeechEmoClassificationService:1.4.20200520190222_C124DE' saved to: C:\Users\Jesudas/bentoml/repository/SpeechEmoClassificationService\1.4.20200520190222_C124DE [2020-05-20 19:02:38,582] DEBUG - BentoML in debug mode, keeping temp directory "C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-xfoo3l_o" C:\Users\Jesudas/bentoml/repository/SpeechEmoClassificationService\1.4.20200520190222_C124DE Traceback (most recent call last): File "C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/inference/spemoclass_service_pack.py", line 38, in print(bento_svc.predict({'file' : 'C:/SL_Projects/SpeechEmotionRecognition/SpeechEmoReco_HireAi/test_wav_samples/RECORDED/kamal_m_ang.wav'})) File "C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py", line 52, in predict predictions = self.artifacts.model.predict_classes(features, batch_size=16) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\bentoml\artifact\keras_model_artifact.py", line 278, in predict_classes return self.keras_model.predict_classes(*args, *kwargs) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\engine\sequential.py", line 267, in predict_classes proba = self.predict(x, batch_size=batch_size, verbose=verbose) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\engine\training.py", line 1169, in predict steps=steps) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\engine\training_arrays.py", line 294, in predict_loop batch_outs = f(ins_batch) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in call return self._call(inputs) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(array_vals) File "C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\tensorflow\python\client\session.py", line 1458, in call run_metadata_ptr) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dense_2/bias [[{{node dense_2/bias/read}}]]

Process finished with exit code 1

parano commented 4 years ago

hi @jesudasvf, it looks like this might be an issue with tensorflow-gpu https://github.com/keras-team/keras/issues/5427, I'd love to help you debug the issue, could you ping me in the bentoml slack channel?

jesudasvf commented 4 years ago

Thanks @Chaoyu, I was able to make bentoml work with my Keras Speech emotion classifier model. I did a manual visual inspection of the example benotml Keras text classification and my classifier code. I was able to find the differences and one difference in the import statement solved the problem. I was able to serve the SpeechEmoClassifierService and connect to it with Rest APIs using the swagger tool. Bentoml_issue_677.pdf

The output of my classifier is 16 class SoftMax output.

[2020-05-28 16:57:12,278] WARNING - auto_pip_dependencies enabled, it may override package versions specified in pip_dependencies=['tensorflow-gpu==1.14.0', 'numpy', 'librosa', 'sklearn', 'keras==2.2.4'] Loading the classifier model : ../saved_models/gender_emotion/SpeechAllGenderEmotionRecognition_2D_M.h5 .... 2020-05-28 16:57:12.608190: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 ../saved_models/gender_emotion/SpeechAllGenderEmotionRecognition_2D_M.h5 classifier model loaded ! WARNING:tensorflow:From C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\bentoml\artifact\keras_model_artifact.py:134: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[[2.2981223e-05 6.1333367e-07 2.7598476e-06 7.3969923e-06 1.3689848e-04 1.1032406e-06 5.7545613e-10 1.0761723e-06 9.7340333e-01 2.6941398e-06 1.1886099e-04 6.8937259e-04 1.3197271e-05 1.5912860e-07 2.5579931e-02 1.9744002e-05]] [2020-05-28 16:57:13,588] DEBUG - Creating local YataiService instance [2020-05-28 16:57:13,748] DEBUG - Upgrading tables to the latest revision [2020-05-28 16:57:13,748] DEBUG - Created temporary directory: C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-mbpjr5y9 [2020-05-28 16:57:16,228] DEBUG - HTTPSConnectionPool(host='api.amplitude.com', port=443): Read timed out. (read timeout=1) [2020-05-28 16:57:16,378] DEBUG - copy_used_py_modules target_module_name: inference.SpeechEmoClassificationService, target_module_file: C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py [2020-05-28 16:57:16,383] DEBUG - Searching for dependant modules of inference.SpeechEmoClassificationService:C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py [2020-05-28 16:57:29,398] DEBUG - Ignoring deps within local site-packages path: ['C:\SL_Projects\SpeechEmotionRecognition\spemoenv\Scripts\python36.zip', 'C:\Python368\lib', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\win32', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\win32\lib', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\Pythonwin', 'C:\Python368\lib\site-packages', 'C:\SL_Projects\SpeechEmotionRecognition\spemoenv', 'C:\Python368'] [2020-05-28 16:57:29,398] DEBUG - User local module deps found: main [2020-05-28 16:57:29,408] DEBUG - Copying user local python dependencies: {'inference.SpeechEmoClassificationService': <module 'inference.SpeechEmoClassificationService' from 'C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py'>} [2020-05-28 16:57:29,408] DEBUG - Copying local python module 'C:\SL_Projects\SpeechEmotionRecognition\SpeechEmoReco_HireAi\inference\SpeechEmoClassificationService.py' [2020-05-28 16:57:29,408] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-mbpjr5y9\SpeechEmoClassificationService' [2020-05-28 16:57:29,408] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-mbpjr5y9\SpeechEmoClassificationService\artifacts' [2020-05-28 16:57:29,408] DEBUG - Creating empty init.py under folder:'C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-mbpjr5y9\SpeechEmoClassificationService\inference' [2020-05-28 16:57:29,408] DEBUG - Done copying local python dependant modules [2020-05-28 16:57:29,603] INFO - BentoService bundle 'SpeechEmoClassificationService:1.4.20200528165713_5C8CA1' saved to: C:\Users\Jesudas/bentoml/repository/SpeechEmoClassificationService\1.4.20200528165713_5C8CA1 [2020-05-28 16:57:29,603] DEBUG - BentoML in debug mode, keeping temp directory "C:\Users\Jesudas\AppData\Local\Temp\bentoml-temp-mbpjr5y9" C:\Users\Jesudas/bentoml/repository/SpeechEmoClassificationService\1.4.20200528165713_5C8CA1 [[2.2981223e-05 6.1333367e-07 2.7598476e-06 7.3969923e-06 1.3689848e-04 1.1032406e-06 5.7545613e-10 1.0761723e-06 9.7340333e-01 2.6941398e-06 1.1886099e-04 6.8937259e-04 1.3197271e-05 1.5912860e-07 2.5579931e-02 1.9744002e-05]] [2020-05-28 16:57:30,958] WARNING - Module inference.SpeechEmoClassificationService already loaded, using existing imported module. WARNING:tensorflow:From C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\bentoml\artifact\keras_model_artifact.py:148: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\tensorflow\python\ops\init_ops.py:97: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From C:\SL_Projects\SpeechEmotionRecognition\spemoenv\lib\site-packages\tensorflow\python\ops\init_ops.py:97: calling Zeros.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor [2020-05-28 16:57:32,808] DEBUG - HTTPSConnectionPool(host='api.amplitude.com', port=443): Read timed out. (read timeout=1) [[2.2981223e-05 6.1333367e-07 2.7598476e-06 7.3969923e-06 1.3689848e-04 1.1032406e-06 5.7545613e-10 1.0761723e-06 9.7340333e-01 2.6941398e-06 1.1886099e-04 6.8937259e-04 1.3197271e-05 1.5912860e-07 2.5579931e-02 1.9744002e-05]]

Process finished with exit code 0

parano commented 4 years ago

Thanks for the detailed notes @jesudasvf - your import statement is actually expected to work, the KerasModelArtifact in bentoml works with both tf.kears and keras. Could you help verify if there's a _keras_module_name.txt file under the saved bundle created? If so, what is the content?

You do need to ensure the Keras module is consistent - if you pack the model with tf.keras, it must be loaded with tf.keras as well, and vise versa.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.