Tensorflow 1.12 model LookupTable initialization error

[ravimadhusudhan8]

Applying application.yaml ... {'kafkaStreaming': [], 'executionGraph': {'stages': [{'services': [{'weight': 100, 'signatureName': 'signature-to-be-ignored', 'runtimeId': 6, 'modelVersionId': 17}]}]}, 'name': 'seq2seq-service'} Server app response {'name': 'seq2seq-service', 'kafkaStreaming': [], 'id': 9, 'executionGraph': {'stages': [{'services': [{'runtime': {'name': 'hydrosphere/serving-runtime-tensorflow', 'tags': ['tensorflow'], 'version': '1.7.0-latest', 'id': 6, 'configParams': {}, 'suitableModelType': ['tensorflow:1.7.0']}, 'modelVersion': {'model': {'name': 'seq2seq', 'modelContract': 'model_name: "seq2seq"\nsignatures {\n signature_name: "serving_default"\n inputs {\n name: "input_data"\n shape {\n dim {\n size: -1\n name: ""\n }\n dim {\n size: 1\n name: ""\n }\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n outputs {\n name: "output"\n shape {\n dim {\n size: -1\n name: ""\n }\n dim {\n size: 1\n name: ""\n }\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n}\n', 'modelType': 'tensorflow:1.12.0', 'id': 12, 'updated': '2019-01-08T23:18:20.951', 'created': '2019-01-08T21:04:12.152'}, 'modelContract': 'model_name: "seq2seq"\nsignatures {\n signature_name: "serving_default"\n inputs {\n name: "input_data"\n shape {\n dim {\n size: -1\n name: ""\n }\n dim {\n size: 1\n name: ""\n }\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n outputs {\n name: "output"\n shape {\n dim {\n size: -1\n name: ""\n }\n dim {\n size: 1\n name: ""\n }\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n}\n', 'imageName': 'seq2seq', 'imageTag': '3', 'modelName': 'seq2seq', 'modelType': 'tensorflow:1.12.0', 'id': 17, 'imageSHA256': 'eaff8f33c808469a48c98fc83a4b0971bff945f9ed0811e2b226119bc0a631e4', 'modelVersion': 3, 'created': '2019-01-08T23:18:23.041'}, 'environment': {'id': 0, 'name': 'Without Env', 'placeholders': []}, 'weight': 100}], 'dataProfileFields': {}}]}, 'contract': 'model_name: "seq2seq"\nsignatures {\n signature_name: "serving_default"\n inputs {\n name: "input_data"\n shape {\n dim {\n size: -1\n name: ""\n }\n dim {\n size: 1\n name: ""\n }\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n outputs {\n name: "output"\n shape {\n dim {\n size: -1\n name: ""\n }\n dim {\n size: 1\n name: ""\n }\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n}\n'} {'application.yaml': [{'contract': 'model_name: "seq2seq"\n' 'signatures {\n' ' signature_name: "serving_default"\n' ' inputs {\n' ' name: "input_data"\n' ' shape {\n' ' dim {\n' ' size: -1\n' ' name: ""\n' ' }\n' ' dim {\n' ' size: 1\n' ' name: ""\n' ' }\n' ' unknown_rank: false\n' ' }\n' ' dtype: DT_STRING\n' ' }\n' ' outputs {\n' ' name: "output"\n' ' shape {\n' ' dim {\n' ' size: -1\n' ' name: ""\n' ' }\n' ' dim {\n' ' size: 1\n' ' name: ""\n' ' }\n' ' unknown_rank: false\n' ' }\n' ' dtype: DT_STRING\n' ' }\n' '}\n', 'executionGraph': {'stages': [{'dataProfileFields': {}, 'services': [{'environment': {'id': 0, 'name': 'Without ' 'Env', 'placeholders': []}, 'modelVersion': {'created': '2019-01-08T23:18:23.041', 'id': 17, 'imageName': 'seq2seq', 'imageSHA256': 'eaff8f33c808469a48c98fc83a4b0971bff945f9ed0811e2b226119bc0a631e4', 'imageTag': '3', 'model': {'created': '2019-01-08T21:04:12.152', 'id': 12, 'modelContract': 'model_name: ' '"seq2seq"\n' 'signatures ' '{\n' ' ' 'signature_name: ' '"serving_default"\n' ' ' 'inputs ' '{\n' ' ' 'name: ' '"input_data"\n' ' ' 'shape ' '{\n' ' ' 'dim ' '{\n' ' ' 'size: ' '-1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'dim ' '{\n' ' ' 'size: ' '1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'unknown_rank: ' 'false\n' ' ' '}\n' ' ' 'dtype: ' 'DT_STRING\n' ' ' '}\n' ' ' 'outputs ' '{\n' ' ' 'name: ' '"output"\n' ' ' 'shape ' '{\n' ' ' 'dim ' '{\n' ' ' 'size: ' '-1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'dim ' '{\n' ' ' 'size: ' '1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'unknown_rank: ' 'false\n' ' ' '}\n' ' ' 'dtype: ' 'DT_STRING\n' ' ' '}\n' '}\n', 'modelType': 'tensorflow:1.12.0', 'name': 'seq2seq', 'updated': '2019-01-08T23:18:20.951'}, 'modelContract': 'model_name: ' '"seq2seq"\n' 'signatures ' '{\n' ' ' 'signature_name: ' '"serving_default"\n' ' ' 'inputs ' '{\n' ' ' 'name: ' '"input_data"\n' ' ' 'shape ' '{\n' ' ' 'dim ' '{\n' ' ' 'size: ' '-1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'dim ' '{\n' ' ' 'size: ' '1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'unknown_rank: ' 'false\n' ' ' '}\n' ' ' 'dtype: ' 'DT_STRING\n' ' ' '}\n' ' ' 'outputs ' '{\n' ' ' 'name: ' '"output"\n' ' ' 'shape ' '{\n' ' ' 'dim ' '{\n' ' ' 'size: ' '-1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'dim ' '{\n' ' ' 'size: ' '1\n' ' ' 'name: ' '""\n' ' ' '}\n' ' ' 'unknown_rank: ' 'false\n' ' ' '}\n' ' ' 'dtype: ' 'DT_STRING\n' ' ' '}\n' '}\n', 'modelName': 'seq2seq', 'modelType': 'tensorflow:1.12.0', 'modelVersion': 3}, 'runtime': {'configParams': {}, 'id': 6, 'name': 'hydrosphere/serving-runtime-tensorflow', 'suitableModelType': ['tensorflow:1.7.0'], 'tags': ['tensorflow'], 'version': '1.7.0-latest'}, 'weight': 100}]}]}, 'id': 9, 'kafkaStreaming': [], 'name': 'seq2seq-service'}]}

[KineticCookie]

Hello This could be the result of runtime initialization failure. Can you check the output of docker ps -a for any Failed or Finished containers?

[ravimadhusudhan8]

Thanks for the suggestion.

I have attached 3 screenshots -

1. On creation of Application - docker ps -a -> shows TF runtime created

2. Test the new Application -> error message shown

3. Again run docker ps -a -> now shows that TF runtime has exited

[KineticCookie]

Yeah, an error happened in the TF container. Can you provide it's logs?

[ravimadhusudhan8]

Please find the screenshots of the logs

1. Just the container name
2. Logs 1
3. Logs 2

[KineticCookie]

🤔 seems like your model has incompatible graph nodes with tf version 1.7.0. There is gonna be newer versions of TF runtimes soon, however I'm not sure if it will fix it. I'll investigate this behavior.

[ravimadhusudhan8]

Thanks