BatchTransform Seq2Seq example "Unable to evaluate payload provided"

[nikhilraju]

Issue Description

The Seq2Seq example notebook has the following step "Using Protobuf format for inference (Suggested for efficient bulk inference)" with the following snippet

# Convert strings to integers using source vocab mapping. Out-of-vocabulary strings are mapped to 1 - the mapping for <unk>
sentences = [[source.get(token, 1) for token in sentence.split()] for sentence in sentences]
f = io.BytesIO()
for sentence in sentences:
    record = list_to_record_bytes(sentence, [])
    write_recordio(f, record)

response = runtime.invoke_endpoint(EndpointName=endpoint_name, 
                                   ContentType='application/x-recordio-protobuf', 
                                   Body=f.getvalue())

response = response["Body"].read()

However, when I try running it, I get an error with the message "ERROR:root:Unable to evaluate payload provided"

Details

In the cloud watch log stream, I see the following additional details in the trace:

Traceback (most recent call last):
File "/opt/amazon/lib/python3.4/site-packages/seq2seq/serve.py", line 246, in invocations
response_payload, mimetype = ScoringService.render_output(content_type, cur_translator, payload)
File "/opt/amazon/lib/python3.4/site-packages/seq2seq/serve.py", line 155, in render_output
return parse_proto_request(model, payload)
File "/opt/amazon/lib/python3.4/site-packages/seq2seq/serve.py", line 344, in parse_proto_request
target_sentences = _recordio_protobuf_encoder(_translate_proto_file(translator, output_file))
File "/opt/amazon/lib/python3.4/site-packages/seq2seq/serve.py", line 313, in _recordio_protobuf_encoder
for results in data:

On further investigation, I found that in the doc for Seq2Seq, it is mentioned that

For batch transform, inference supports JSON Lines format. Batch transform expects the input in JSON Lines format and returns the output in JSON Lines format. Both content and accept types should be application/jsonlines. The format for input is as follows:

And I noticed that in the example notebook the step used ContentType='application/x-recordio-protobuf for Batch transform.

Am I missing something? Or should the example notebook be updated to use content type application/jsonlines?

Thanks!

[orchidmajumder]

Hi, I apologize for the confusion. The notebook is not updated to demonstrate how to use Batch Transform feature for SageMaker. The bulk inference feature mentioned here was the way to do bulk inference before SageMaker launched Batch Inference. This is not the recommended way right now, we should be updating the notebook.

For Batch Transform, your input data should reside in S3 and you should use using the create_transform_job API (or sagemaker.transformer.Transformer if you are using Python SDK instead of Boto3) to create a Batch Transform job. There is no need to create an endpoint and you can continue to use your existing Model.

The input to the Batch Transformation job (which should be in S3) should have the contents organized as mentioned in the documentation you found out and content-type should be passed as application/jsonlines.

Here is one example (on a different algorithm though) on how to use Batch Transform using the same Model created for online inference (Batch Transform is at the end) : https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/inference_pipeline_sparkml_blazingtext_dbpedia/inference_pipeline_sparkml_blazingtext_dbpedia.ipynb