cHttpRequest is duplicating the first ~3389 bytes of XML response

[Montclair]

Using mdTlsThunks cAsyncSocket cHttpRequest cTlsSocket OS: XP POS Ready

I'm using an API at icecat.biz to download product specifications. It requires a username and password so I'm not sure how far you can get testing -- but sign up is free. Regardless, using cHttpRequest, the first 3389 to 3390 bytes are getting duplicated in the .ResponseText. It's showing up like this

</FeatureGroup>
  </Cate<?xml version="1.0" ... where the rest of this is correct and is the full XML document

If I search the .ResponseText for "<?xml version" after the first 15 bytes, I'm able to locate it and truncate the duplication, as a workaround.

Lastly, calls are failing frequently with no response at all, and no error thrown. I sent you a PM about this on vbforums.com with more data and a URL to test with login and password. Maybe 1 out of 5 calls works properly at least in the IDE. I'm not sure if something isn't getting closed, even though I'm setting the object to nothing after the call, or what exactly is happening.

Sorry to be so vague here but I wanted to alert you at least to the issue here in case you missed it at vbforums.com

---

Edit: One of your routines is throwing a subscript out of range during the times that the connection fails. It's difficult to nail down in what routine this is happening though, but it happens during the .Send call.

[Montclair]

Just to add to the issue, I'm also getting incomplete responses. It may be related to the above. Here's sample code that doesn't return the server's complete response:

    Dim objhttp As cHttpRequest
    Set objhttp = New cHttpRequest

    With objhttp       

        'Allow redirects
        .Option_(WinHttpRequestOption_EnableRedirects) = True
        'Enable Https To Http Redirects
        .Option_(WinHttpRequestOption_EnableHttpsToHttpRedirects) = True
        'Enable Http 1.1
        .Option_(WinHttpRequestOption_EnableHttp1_1) = True
        'Ignore all certificate errors
        .Option_(WinHttpRequestOption_SslErrorIgnoreFlags) = SslErrorFlag_Ignore_All
        'Allow 15 seconds for everything to do what it has to do
        .SetTimeouts 15000, 15000, 15000, 15000
        .Open_ "GET", "https://clienttest.ssllabs.com:8443/ssltest/viewMyClient.html", False
        .SetRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0"
        .SetRequestHeader "Content-type", "text/html"
        .Send
        Debug.Print "RESPONSE LENGTH: "; Len(.ResponseText)
        Debug.Print "RESPONSE: "; .ResponseText
    End With

    Set objhttp = Nothing

In this example, I only get either 7143 or 8192 bytes. If I do that with straight WinHttp, the response length is always 19571 bytes.

[wqweto]

Oops, this should be fixed in b4c25fa53874e12a305a0bea77ad7f6a9219b21f now

10x for reporting!

[Montclair]

Getting Subscript out of Range raised in cHttpReuest's Send subroutine. m_vLastError(1) = cHttpRequest.pvRecvBody

Tracing the code, the error is being generated in pvArrayWriteBlob, with this line being the offender:

Call CopyMemory(baArray(lPos), ByVal lPtr, lSize)

This happening with a call to icecat.biz's API that I PM'd you about on VBForums.

[wqweto]

Does clienttest.ssllabs.com:8443 work for you now?

[Montclair]

Yes. However, the call to that XML API I PMd you is generating an error now. More info on that

lpos = -1, which is causing the error lptr = 83849144 lsize = 4096

[wqweto]

This works now

Private Sub pvTestIcecat()
    Dim objhttp As cHttpRequest
    Set objhttp = New cHttpRequest

    With objhttp
        .SetTimeouts 5000, 5000, 5000, 150000
'        .Open_ "GET", "https://#####@data.icecat.biz/xml_s3/xml_server3.cgi?prod_id=%42%4E%55%43%31%31%54%4E%4B%49%35%30%30%30%31;vendor=%49%6E%74%65%6C;lang=US ;output=productxml;shopname=#####"
        .Open_ "GET", "https://#####@data.icecat.biz/xml_s3/xml_server3.cgi?prod_id=BNUC11TNKI50001;vendor=Intel;lang=US%20;output=productxml;shopname=#####"
        .SetRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0"
        .SetRequestHeader "Content-type", "text/html"
        .Send
        Debug.Print .Status; " "; .StatusText
        Debug.Print Len(.ResponseText)
    End With
End Sub

Note that with URL escaped the API returns 400 bad request.

[Montclair]

You have an extra space in the encoded URL. Removing that and it functions identically. lang=US--> <--;output=productxml

have to remove the space after US

It doesn't work reliably here. Sometimes it does, most times it throws subscript out of range. Happens in the unencoded URL as well just to eliminate that as a possible error.

Try it several times and see if it doesn't fail for you.

[Montclair]

Oops, sorry. Didn't see the new commit. Trying that and will report back.

[Montclair]

Working! Excellent! Thanks!

One aside -- I was also using this to download images from the same resource. Many of the images were coming through as incomplete, but some were good. Do you suspect that these bugs were the cause?

[wqweto]

Can you write a repro? Might be a simple loop which downloads images until failing

[Montclair]

It's basically identical code with this stuff tacked on:

Dim b() As Byte, FF as Integer

.... .Open_ "GET", Url, False .SetRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Firefox/52.0" .SetRequestHeader "Content-type", "text/html" .Send b() = .ResponseBody End With

If UBound(b) > 2 Then
    Open SaveAsPath & Filename For Binary Access Write As #FF
    Put #FF, , b()
    Close #FF
End If

For whatever reason, the icecat image server doesn't use the same ciphers so I can still connect to that one with winhttp so I just reverted the code. I don't have time ATM to put it back and run a bunch of tests, but I will try to do so later today or tomorrow and report back. I do suspect the fixes you made today fixed that problem as well, though.

Thanks, again!

[wqweto]

You could try changing content type to image/png or application/octet-stream -- something binary.

Btw, probably images are served from a CDN so using different setup for https.

[wqweto]

I just did some tests here with downloading a bunch of PNG and couldn't reproduce any issues with binary transfers.

Here is the test code

Private Sub pvTestBinary()
    Const STR_REMOTE    As String = "https://dl.unicontsoft.com/upload/pix/"
    Const STR_LOCAL     As String = "D:\TEMP\PNG\"
    Dim vElem           As Variant
    Dim baRemote()      As Byte
    Dim baLocal()       As Byte

    With New cHttpRequest
        For Each vElem In EnumFiles(STR_LOCAL)
            '--- strip filename only
            vElem = Mid$(vElem, InStrRev(vElem, "\") + 1)
            .Open_ "GET", STR_REMOTE & vElem
            .SetRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0"
            .SetRequestHeader "Content-type", "text/html"
            .Send
            baRemote = .ResponseBody
            baLocal = ReadBinaryFile(STR_LOCAL & vElem)
            If UBound(baLocal) <> UBound(baRemote) Then
                Debug.Print vElem & ": size differs", UBound(baLocal), UBound(baRemote)
            ElseIf InStrB(1, baLocal, baRemote) <> 1 Then
                Debug.Print vElem & ": content differs"
            End If
        Next
    End With
End Sub

Will test on OS: XP POS Ready later if OS makes any difference.

[Montclair]

That's great. I didn't get around to testing it yesterday. I was pretty sure you squashed that bug by fixing the other issues yesterday. Great work! I think you can close this issue. It's all working very well on my end now. I'm going to mess around with creating a .dll from this so I can call it from VBA applications. Thanks!

[Montclair]

Happy to report that I was able to create a DLL without typing a line of code, and rolled this out to my VBA applications that were using WinHTTP, making only very minimal changes to them and it works flawlessly. This project of yours is a lifesaver! +1000!

[wqweto]

Sure!

I removed it from shopname in query string too.

[Montclair]

Yep. Thanks!