Is there a way to output binary data from response body?

[dreamercz]

I have the following test scenario that GETs a zip file as a binary stream in the response body. I am trying to save the binary stream as a zip file again. The below code does not fail and the zip file is saved, but I can not open the file.

Library    REST    ${HOST}
Library    OperatingSystem

Test scenario
    GET    ${base_path}foo/
    Integer    response status    200
    ${response} =    Output    response body
    ${b_response} =    Encode String To Bytes    ${response}    utf_8
    Create Binary File    ${OUTPUTDIR}/files/test.zip    ${b_response}

The Output for response body is the following, I copy paste it verbatim from the log:

'PK\x03\x04-\x00\x08\x08\x08\x00&�MS\x00\x00\x00\x00��������\x13 [truncated]

I think this might be happening because Output treats the value as a string. Is there a better approach to all this with RESTInstance?

[asimell]

If converting string to bytes doesn't work, then I don't think it's straight doable with RESTinstance. We can investigate if we could add support for different data types easily and add that to the next release.

One option would be to enable writing this straight to a file. This could potentially work somehow like this

# Note, currently there's no `write_mode` argument
Output    response body    file_path=my_file    write_mode=bytes
Output    response body    file_path=my_file    write_mode=string   # default, like now

[Atihinen]

It seems that the robot own libraries are not working correctly (Encode String To Bytes or Create Binary File). Can you create issue to https://github.com/robotframework/robotframework/issues ?

[dreamercz]

Hi, thanks for looking at it!

I'm not sure that it's a problem with the standard library, I tried the same with RequestsLibrary and it worked OK.

${response} =    RequestsLibrary.GET    ${url}
    ...    headers=${headers}
    ...    expected_status=200
    Create Binary File    ${OUTPUTDIR}/files/${save_as}    ${response.content}

[asimell]

RequestsLibrary returns a Response object instead of a JSON (as this library does). Also response.content returns the value of the content in bytes. With RESTinstance you need to encode that to bytes with Encode String To Bytes before you can write it to a binary file. So, there is probably something weird happening when you convert your bytes into string, so it's usable by RESTinstance and then back to bytes with Encode String To Bytes.

[dreamercz]

You're right, I didn't realize that.

I'll try to find the time to have a closer look at the behavior and then submit a new issue to RF.

[asimell]

Also, depending on your ${headers} you might also need to change your headers with RESTinstance. By default it accepts application/json, so you might need to change that to another value for RESTinstance to handle the data properly.

[dreamercz]

Sorry for the delay. Here's what I found so far.

It seems like the issue stems from Requests creating the Unicode string for response.text ie. what RESTInstance falls back to when the response is not a valid JSON src/REST/keywords.py#L1382

So I did a little experiment. I went to replicate what I was trying to do in Robot with just the Requests lib That is, send a GET requests, in the response obtain some binary data, save those as a file.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from requests import get

response = get("https://cataas.com/cat")
response.encoding = 'utf-8'

# creates a broken file "Error interpreting JPEG image file (Not a JPEG file: starts with 0xef 0xbf)"
text = response.text
encoded = text.encode()
with open('as_text.jpg', 'wb') as f:
    f.write(encoded)

# this creates the jpg image with no problem
with open('as_content.jpg', 'wb') as f:
    f.write(response.content)

If I take the binary content of the response as is, I can save it to a file with no problems. However, this doesn't work if I try to convert the Unicode string Requests gives me through the text attribute.

IMO there is something weird going on with either the conversion to Unicode from the Requests lib or with the conversion back to bytes using the std lib function. There is also the possibility I am doing it all wrong, I haven't discounted that yet. :grin:

[asimell]

@dreamercz Does your response have the "Accept-Ranges": "bytes" header? We're considering our options if we want to support giving both response.text and response.content in our response, somehow parse the Content-Type header to identify which we should use, identify other possibilities, or decide that we're not going to support this.

The "Accept-Ranges": "bytes" could/would significantly reduce the amount of work needed for this, if we could just check if that header is present, then provide response.content and otherwise just give response.text as is the current behaviour.

[asimell]

Closing due to lack of knowledge. Please reopen if still relevant.