Closed IngwiePhoenix closed 8 years ago
Oh by the way, error only occurs on HTTPS - never on HTTP.
I will look into this fully when i have time, but won't have time until early april unfortunately...
From a quick glance google.com does a weird redirect sorta thing (sets a cookie for google.at in my case). I get the following line of trace print: "_ParseHeader: Not chunked transfer and content-length==0, this will go fail". Explains why it crashes. But i have no idea why google is sending this weird HTTP reply. Will push something real quick to fix the crash at least, but a better fix has to wait until later.
What. google.com sends this in the header:
alternate-protocol: 443:quic,p=1
alt-svc: quic=":443"; ma=2592000; v="31,30,29,28,27,26,25"``
Maybe there is a special header that has to be sent that quic is not supported but i can't imagine that, since HTTP is all about backwards compatibility. Seems to be an issue specific to google, at least it didn't happen for me with any other site i tested. Might be related to a mbedtls misconfiguration on client side that google is specifically detecting, but i doubt this.
What really confuses me is that google replies with some headers but an empty body (which is why neither content-length not chunked transfer encoding is set. Makes sense but i haven't seen this sort of reply before.)
Right now i have no idea how to make the server happy to send an actual body. Try to fake a user-agent of a popular browser? Just did a cross-check with curl and that works.
I sadly didn't find out a good way to get Google to work, but found out in the process that https://api.github.com/zen
also does not work - it results in the contents being NULL
, triggering the "Error!" message in example 1. Any idea for that one?
Actually nevermind... that was totally an issue on my end. I ended up getting it working. :)
I had to set a User-Agent here as well. Example code:
// Example 1: Dead-simple one-shot file download API
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include "minihttp.h"
using namespace minihttp;
class DLSocket : public HttpSocket
{
public:
DLSocket() : buf(NULL), bufsz(0), bufcap(0), finished(false), fail(false){}
virtual ~DLSocket() {}
char *buf;
size_t bufsz;
size_t bufcap;
bool finished;
bool fail;
protected:
void _OnRequestDone()
{
finished = true;
buf[bufsz] = 0; // zero-terminate
}
void _OnRecv(void *incoming, unsigned size)
{
if(!size || !IsSuccess())
return;
if(bufcap + size + 1 >= bufsz) // always make sure there's 1 more byte free for the zero-terminator
{
bufcap += (bufcap / 2) + size + 1;
buf = (char*)realloc(buf, bufcap);
if(!buf)
{
fail = true;
close();
}
}
memcpy(buf + bufsz, incoming, size);
bufsz += size;
}
};
int main(int argc, char *argv[]){
char* url = "https://api.github.com/zen";
if(!InitNetwork())
return 2;
DLSocket dl;
dl.SetBufsizeIn(64 * 1024);
dl.SetNonBlocking(false);
dl.SetFollowRedirect(true);
dl.SetAlwaysHandle(false);
//dl.SetUserAgent("curl/7.40.0");
dl.Download(url, NULL, NULL, NULL);
while(dl.isOpen() || dl.HasPendingTask())
dl.update();
if(!dl.finished || dl.fail)
{
free(dl.buf);
return 2;
}
puts(dl.buf);
free(dl.buf);
return 0;
}
Consider giving the minihttp::Download
api an argument to set the user-agent?
No arg, just a default. That function shouldn't be used anyway ;) Thanks for the investigation so far.
EDIT: I have no idea why (or if) the User-Agent field is required. Don't think the spec mandates it...
This should explain it all:
I mimiced the mbedtls API in the scripting language, and
-e
executes a string. From thereon out, you should see where this is going :)