Error on Mac when parsing redirected response

[bergera]

On Mac, WebResponse.web_ExtractResponseTextFromCurlResponse() improperly handles responses which get redirected.

In this example, the raw output of curl includes the HTTP 301 response, followed by a blank line, followed by the desired HTTP 200 response, followed by a blank line, then the reponse body. web_ExtractResponseTextFromCurlResponse then returns everything after the first blank line as the response content, which includes HTTP headers.

This silly example is solved simply by adding another / to the end of the URL, and our actual use case probably won't involve redirects, but I thought it worthwhile to report.

Thanks for this great library, it solves a lot of problems for us!

Example Code

Function GetPokemon(Id as Integer) As String
    Dim PokeClient As New WebClient
    PokeClient.BaseUrl = "http://pokeapi.co/api/v2/"

    Dim Resource As String
    Dim Response As WebResponse

    Resource = "pokemon/" & Id

    Set Response = PokeClient.GetJson(Resource)

    ProcessPokemon Response
End Function

Public Sub ProcessPokemon(Response As WebResponse)
    If Response.StatusCode = WebStatusCode.Ok Then
        Dim Pokemon As Dictionary
        Set Pokemon = Response.Data

        Debug.Print "Pokemon's name is " & Pokemon("species")("name")
    Else
        Debug.Print "Error: " & Response.Content
    End If
End Sub

Raw curl Response

$ curl -i --connect-timeout 5 --max-time 15 --location -H 'User-Agent: VBA-Web v4.1.1 (https://github.com/VBA-tools/VBA-Web)' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Content-Length: 0' -X GET -d '' 'http://pokeapi.co/api/v2/pokemon/50'

HTTP/1.1 301 MOVED PERMANENTLY
Date: Mon, 19 Sep 2016 18:57:03 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=df7d96817b162680b6e444f5f79e4e2be1474311423; expires=Tue, 19-Sep-17 18:57:03 GMT; path=/; domain=.pokeapi.co; HttpOnly
X-Frame-Options: SAMEORIGIN
Location: http://pokeapi.co/api/v2/pokemon/50/
X-XSS-Protection: 1; mode=block
Server: cloudflare-nginx
CF-RAY: 2e4f42dcb4fb54ce-ORD

HTTP/1.1 200 OK
Date: Mon, 19 Sep 2016 18:57:04 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d8b746500be10ce133918f63958a8aa311474311423; expires=Tue, 19-Sep-17 18:57:03 GMT; path=/; domain=.pokeapi.co; HttpOnly
Vary: Accept-Encoding
Vary: Cookie
X-Frame-Options: SAMEORIGIN
Allow: GET, HEAD, OPTIONS
X-XSS-Protection: 1; mode=block
Server: cloudflare-nginx
CF-RAY: 2e4f42dda30610bd-ORD

{"forms":[{"url":"http:\/\/pokeapi.co\/api\/v2\/pokemon-form\/1\/","name":"bulbasaur"}],"abilities":

Error Message

ERROR - WebHelpers.ParseByFormat: 11000, An error occurred during parsing
10001: Error parsing JSON:
HTTP/1.1 20
^
Expecting '{' or '['
ERROR - WebHelpers.ParseByFormat: 11000, An error occurred during parsing
10001: Error parsing JSON:
HTTP/1.1 20
^
Expecting '{' or '['
Error: HTTP/1.1 200 OK
Date: Mon, 19 Sep 2016 17:23:11 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d121f6cd355ec8758348dfa6931a947471474305790; expires=Tue, 19-Sep-17 17:23:10 GMT; path=/; domain=.pokeapi.co; HttpOnly
Vary: Accept-Encoding
Vary: Cookie
X-Frame-Options: SAMEORIGIN
Allow: GET, HEAD, OPTIONS
X-XSS-Protection: 1; mode=block
Server: cloudflare-nginx
CF-RAY: 2e4eb95627cd257f-ORD

{"forms":[{"url":"http:\/\/pokeapi.co\/api\/v2\/pokemon-form\/1\/","name":"bulbasaur"}],"abilities":

[ghost]

I was about to post the same thing.

The WebResponse class needs updates to properly locate

• The final HTTP status code
• The body of the last request

[ghost]

First method that requires attention is the one that retrieve status code:

https://github.com/VBA-tools/VBA-Web/blob/ee6f34de9c534560b145002b5e8a28fdf2eceba7/src/WebResponse.cls#L384-L394

This needs to be made aware of possible redirects, in which case it should keep processing requests until it gets to the last one made.

[ghost]

Probably every method called in this section requires attention:

https://github.com/VBA-tools/VBA-Web/blob/ee6f34de9c534560b145002b5e8a28fdf2eceba7/src/WebResponse.cls#L175-L177

[dyt811]

For posterity reason, I would like to add, that for me, by using httpS instead of http for the endpoint that I want to reach, it got around the redirect/301 issue.

Again, probably not going to magically solve all 301 redirect problem but it should be the first thing to try if you are in similar boat. i.e. try to directly reach the end destination (in my case... the http > https address).

[MubaiHua]

This issue is still unsolved. Is there any solution or still need to manually parse the HTTP response