Make sure the whole response data is read when using urllib2.urlopen

[ahumeau]

The read method does not guarantee to retrieve the whole data from urlopen's result. Using readlines instead will do the trick.

[coleifer]

Can you provide more information on why read() does not read all of the response?

[coleifer]

It looks like the important part of read() is implemented:

            while True:
                try:
                    data = self._sock.recv(rbufsize)
                except error, e:
                    if e.args[0] == EINTR:
                        continue
                    raise
                if not data:
                    break
                buf.write(data)
            return buf.getvalue()

I'm not seeing how this would avoid reading everything.

[ahumeau]

Hi, sorry, I was off the internet those past days. The only reference I found regarding this issue is this Stackoverflow entry. Yet I did experience the problem, making bootstrap_embedly fail when decoding the incomplete JSON data. Using readlines or requests "solved" the issue.

[coleifer]

Is there a particular site or domain that is giving you problems? Can you give me any code to try to replicate the issue? I'm definitely curious what might be going on.