Open echan00 opened 5 years ago
I have met the same problem too. My environments: tensor2tensor 1.9.0 tensorflow_gpu 1.11.0 tensorflow-serving and tensorflow-serving-api 1.11.0 And I have tried using more recent versions of t2t & tf, but it doesn't work!
@leemingtian curious to know which source and target languages your model is serving
@echan00 translate_enzh_wmt32k_rev
I'm wondering if Chinese -> English is causing the problem.
While I was debugging another issue earlier, I noticed in translate_enzh.py line 243 the source and target vocab files were the other way around.
source_vocab_filename = os.path.join(data_dir, self.source_vocab_name)
target_vocab_filename = os.path.join(data_dir, self.target_vocab_name)
translate_enzh_wmt32k_rev should be Chinese to English. But I saw that source_vocab_filename was EN and target_vocab_filename was ZH
@echan00 Yes, I can get correct response when I using translate_enzh_wmt32k. Maybe you are right.
source_vocab_filename = os.path.join(data_dir, self.source_vocab_name) target_vocab_filename = os.path.join(data_dir, self.target_vocab_name)
Looks like the problem isn't the one mentioned above.
@leemingtian could you possibly add a print(request)
command as I have shown below in serving_utils.py (line 113)? I'm interested to see how the request compared between translate_enzh_wmt32k and translate_enzh_wmt32k_rev
def _make_grpc_request(examples):
"""Builds and sends request to TensorFlow model server."""
request = predict_pb2.PredictRequest()
request.model_spec.name = servable_name
request.inputs["input"].CopyFrom(
tf.contrib.util.make_tensor_proto(
[ex.SerializeToString() for ex in examples], shape=[len(examples)]))
print(request) #print request to see what it looks like
response = stub.Predict(request, timeout_secs)
outputs = tf.make_ndarray(response.outputs["outputs"])
scores = tf.make_ndarray(response.outputs["scores"])
assert len(outputs) == len(scores)
return [{
"outputs": outputs[i],
"scores": scores[i]
} for i in range(len(outputs))]
For translate_enzh_wmt32k_rev the request looks like this:
>> model_spec {
name: "transformer"
}
inputs {
key: "input"
value {
dtype: DT_STRING
tensor_shape {
dim {
size: 1
}
}
string_val: "\n\021\n\017\n\006inputs\022\005\032\003\n\001\001"
}
}
Would appreciate if anybody else can also chime in on this issue!
@echan00 I can get correct response in Chinese -> English now. I have fixed the serving_utils.py as below.
def predict(inputs_list, problem, request_fn):
"""Encodes inputs, makes request to deployed TF model, and decodes outputs."""
assert isinstance(inputs_list, list)
#fname = "inputs" if problem.has_inputs else "targets"
fname = "targets"
input_encoder = problem.feature_info[fname].encoder
input_ids_list = [
_encode(inputs, input_encoder, add_eos=problem.has_inputs)
for inputs in inputs_list
]
examples = [_make_example(input_ids, problem, fname)
for input_ids in input_ids_list]
predictions = request_fn(examples)
#output_decoder = problem.feature_info["targets"].encoder
output_decoder = problem.feature_info["inputs"].encoder
outputs = [
(_decode(prediction["outputs"], output_decoder),
prediction["scores"])
for prediction in predictions
]
return outputs
And then use t2t-query-server with argument "--problem=translate_enzh_wmt32k". (Attention here: argument problem with translate_enzh_wmt32k_rev will not work.)
Hope this helps.
Wow thanks @leemingtian, I'll commit a fix to this repo at some point.
Trying to serve my Chinese to English model and am having trouble querying. I am receiving an error:
The model server seems to be working fine and responding with the same error:
My environment: tensor2tensor (1.10.0) tensorboard (1.12.0) tensorflow (1.12.0) tensorflow-serving-api (1.12.0)
Would appreciate any tips or comments.