tensorflow_model_server sentiment_imdb "Could not parse example input"

ShahNewazKhan commented 6 years ago

Description

I exported a trained transformer model for the sentiment_imdb problem with the following

t2t-exporter \ --model=transformer \ --hparams_set=transformer_small \ --problem=sentiment_imdb \ --data_dir= \ --output_dir=

I then started the tensorflow_model_server as such

tensorflow_model_server   --port=9000   --model_name=babel_sentiment   --model_base_path=/home/shahkhan/t2t_train/sentiment_imdb/transformer-transformer_small/export/Servo  --rest_api_
port=8501
2018-06-19 08:25:23.879161: I tensorflow_serving/model_servers/main.cc:153] Building single TensorFlow model file config:  model_name: babel_sentiment model_base_path: /home/shahkhan/t2t_train/sentiment_imdb/tran
sformer-transformer_small/export/Servo
2018-06-19 08:25:23.879320: I tensorflow_serving/model_servers/server_core.cc:444] Adding/updating models.
2018-06-19 08:25:23.879336: I tensorflow_serving/model_servers/server_core.cc:499]  (Re-)adding model: babel_sentiment
2018-06-19 08:25:23.980120: I tensorflow_serving/core/basic_manager.cc:716] Successfully reserved resources to load servable {name: babel_sentiment version: 1529390674}
2018-06-19 08:25:23.980183: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: babel_sentiment version: 1529390674}
2018-06-19 08:25:23.980215: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: babel_sentiment version: 1529390674}
2018-06-19 08:25:23.980293: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:360] Attempting to load native SavedModelBundle in bundle-shim from: /home/shahkhan/t2t_train/sentiment_imdb/
transformer-transformer_small/export/Servo/1529390674
2018-06-19 08:25:23.980357: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:242] Loading SavedModel with tags: { serve }; from: /home/shahkhan/t2t_train/sentiment_imdb/transformer-transformer_small/
export/Servo/1529390674
2018-06-19 08:25:24.214776: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:161] Restoring SavedModel bundle.
2018-06-19 08:25:24.272312: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:196] Running LegacyInitOp on SavedModel bundle.
2018-06-19 08:25:24.309699: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:291] SavedModel load for tags { serve }; Status: success. Took 328748 microseconds.
2018-06-19 08:25:24.316452: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: babel_sentiment version: 1529390674}
2018-06-19 08:25:24.318322: I tensorflow_serving/model_servers/main.cc:323] Running ModelServer at 0.0.0.0:9000 ...
[evhttp_server.cc : 235] RAW: Entering the event loop ...
2018-06-19 08:25:24.319404: I tensorflow_serving/model_servers/main.cc:333] Exporting HTTP/REST API at:localhost:8501 ...

I ensured default_serving signature is present in the exported model saved_model.pbtxt as such

signature_def {
    key: "serving_default"
    value {
      inputs {
        key: "input"
        value {
          name: "serialized_example:0"
          dtype: DT_STRING
          tensor_shape {
            dim {
              size: -1
            }
          }
        }
      }
      outputs {
        key: "batch_prediction_key"
        value {
          name: "DatasetToSingleElement:0"
          dtype: DT_INT32
          tensor_shape {
            dim {
              size: -1
            }
            dim {
              size: 1
            }
          }
        }
      }
      outputs {
        key: "outputs"
        value {
          name: "transformer/while/Exit_3:0"
          dtype: DT_INT64
          tensor_shape {
            dim {
              size: -1
            }
            dim {
              size: -1
            }
          }
        }
      }
      outputs {
        key: "scores"
        value {
          name: "transformer/while/Exit_12:0"
          dtype: DT_FLOAT
          tensor_shape {
            dim {
              size: -1
            }
          }
        }
      }
      method_name: "tensorflow/serving/predict"
    }
  }

However I get an input parsing error when pinging the REST API as such:

shahkhan@:~$ curl -i -d '{"signature_name": "serving_default","instances": ["this is a great movie"]}' -X POST http://localhost:8501/v1/models/babel_sentiment:predict  -H “Content-Type: application/json”
HTTP/1.1 400 Bad Request
Content-Type: application/json
Date: Tue, 19 Jun 2018 16:29:57 GMT
Content-Length: 652

{ "error": "Could not parse example input, value: \'this is a great movie\'\n\t [[Node: ParseSingleExample/ParseSingleExample = ParseSingleExample[Tdense=[DT_INT64, DT_INT64], dense_keys=[\"batch_prediction_key\", \"targets\"], dense_shapes=[[1], [1]], num_sparse=1, sparse_keys=[\"inputs\"], sparse_types=[DT_INT64]](arg0, ParseSingleExample/Reshape, ParseSingleExample/Const)]]\n\t [[Node: DatasetToSingleElement = DatasetToSingleElement[_output_shapes=[[?,1], [?,?,1,1], [?,1,1,1]], output_shapes=[[?,1], [?,?,1,1], [?,1,1,1]], output_types=[DT_INT32, DT_INT32, DT_INT32], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](MapDataset_4)]]" }curl: (6) Could not resolve host: application

...

Environment information

OS: Ubuntu 16.04

$ pip freeze | grep tensor
tensor2tensor==1.6.3
tensorboard==1.8.0
tensorflow-gpu==1.8.0

$ python -V
Python 3.6.3 :: Anaconda custom (64-bit)

For bugs: reproduction and error logs

# Steps to reproduce:

t2t-exporter \
  --model=transformer \
  --hparams_set=transformer_small \
  --problem=sentiment_imdb \
  --data_dir=<PATH_TO_DATA_DIR> \
  --output_dir=<PATH_TO_TRAIN_DIR>

tensorflow_model_server   --port=9000   --model_name=babel_sentiment   --model_base_path=/home/shahkhan/t2t_train/sentiment_imdb/transformer-transformer_small/export/Servo  --rest_api_
port=8501

curl -i -d '{"signature_name": "serving_default","instances": ["this is a great movie"]}' -X POST http://localhost:8501/v1/models/babel_sentiment:predict  -H “Content-Type: application/json”

# Error logs:
2018-06-19 08:26:07.406673: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input, value: 'this 
was a great movie'
2018-06-19 08:26:07.407138: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at iterator_ops.cc:633 : Invalid argument: Could not parse example input, value: 'this was a g
reat movie'
         [[Node: ParseSingleExample/ParseSingleExample = ParseSingleExample[Tdense=[DT_INT64, DT_INT64], dense_keys=["batch_prediction_key", "targets"], dense_shapes=[[1], [1]], num_sparse=1, sparse_keys=["inputs
"], sparse_types=[DT_INT64]](arg0, ParseSingleExample/Reshape, ParseSingleExample/Const)]]

rgwt123 commented 6 years ago

@ShahNewazKhan I encountered the same problem. Have you solved it?

ShahNewazKhan commented 6 years ago

@rgwt123 I used t2t_query with the appropriate modifications as described in my solution here https://github.com/tensorflow/tensor2tensor/issues/868#issuecomment-399042915

rgwt123 commented 6 years ago

@ShahNewazKhan I follow your example， and get stable output depending on the targets in _make_example. Does it mean if I want to get the correct translation I need to run a loop and do beam search myself?(using transformer) Can you show me how you get the correct translation?

ShahNewazKhan commented 6 years ago

@rgwt123 I am sorry I am not familiar with the language translation properties of the transformer model. The solution I pointed towards is for scoring sentiment on documents as set out in the sentiment_imdb problem.

For sentiment_imdb scoring, you do not need to pass in the targets as the query.py script is reusing the input method from the training interface.

tensorflow / tensor2tensor