Can't read data from the server

RealYukiSan commented 2 weeks ago

I've read in the docs that recv_into will return zero if there's no more received data, but the following recv_into always immediately return 0 at the first call, I wondering is the way I code fundamentally wrong?

import wolfssl
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
secure_sock = wolfssl.wrap_socket(sock)
secure_sock.connect(("dns.google", 443))
secure_sock.write(b"GET / HTTP/1.1\r\nHost: dns.google\r\n\r\n")

response = b""
while secure_sock.recv_into(response, 1024) != 0:
    print("never executed")

print(response)

secure_sock.close()

lealem47 commented 2 weeks ago

Hi @RealYukiSan,

To get this working, you'll have to make response into a mutable buffer. For eg:

response = bytearray(1024)  # A mutable buffer for the recv_into method
data = b""  # To accumulate the received data

while True:
    num_bytes = secure_sock.recv_into(response, 1024)
    if num_bytes == 0:
        break  # Stop if no more data is received
    data += response[:num_bytes]  # Accumulate the data read into 'data'

RealYukiSan commented 2 weeks ago

Thanks @lealem47 for the help, I tried to print num_bytes, it seems work but the num_bytes never reach 0?

$ python test.py 
1024
354
1024
80
5

lealem47 commented 2 weeks ago

That means it never stops receiving data. Could be that the google server is sending you keep-alives

RealYukiSan commented 2 weeks ago

I don't see keep-alive header in the response:

$ python test.py 
bytearray(b'HTTP/1.1 200 OK\r\nContent-Security-Policy: object-src \'none\';base-uri \'self\';script-src \'nonce-m_z8CZg3UI2hfHXAVIjMYw\' \'strict-dynamic\' \'report-sample\'
 \'unsafe-eval\' \'unsafe-inline\' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors \'none\'\r\nStrict-Transport-Security: max-age=315360
00; includeSubDomains; preload\r\nX-Content-Type-Options: nosniff\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Security-Policy-Report-Only: script-src \'none\'; form-
action \'none\'; frame-src \'none\'; report-uri https://csp.withgoogle.com/csp/scaffolding/ntdsgswbsc:55:0\r\nCross-Origin-Opener-Policy-Report-Only: same-origin; report-to=c
oop_reporting\r\nReport-To: {"group":"coop_reporting","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/scaffolding/ntdsgswbsc:55:0"}],}\r\nDate
: Tue, 15 Oct 2024 23:15:33 GMT\r\nServer: scaffolding on HTTPServer2\r\nCache-Control: private\r\nX-XSS-Protection: 0\r\nX-Frame-Options: SAMEORIGIN\r\nAlt-Svc: h3=":443"; m
a=2592000,h3-29=":443"; ma=2592000\r\nAcce')
bytearray(b'pt-Ranges: none\r\nVary: Accept-Encoding\r\nTransfer-Encoding: chunked\r\n\r\n565\r\n<!DOCTYPE html>\n<html lang="en"> <head> <title>Google Public DNS</title>  <m
eta charset="UTF-8"> <link href="/static/93dd5954/favicon.png" rel="shortcut icon" type="image/png"> <link href="/static/e6eca759/matter.min.css" rel="stylesheet"> <link href
="/static/b8536c37/shared.cssent-Type-Options: nosniff\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Security-Policy-Report-Only: script-src \'none\'; form-action \'no
ne\'; frame-src \'none\'; report-uri https://csp.withgoogle.com/csp/scaffolding/ntdsgswbsc:55:0\r\nCross-Origin-Opener-Policy-Report-Only: same-origin; report-to=coop_reporti
ng\r\nReport-To: {"group":"coop_reporting","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/scaffolding/ntdsgswbsc:55:0"}],}\r\nDate: Tue, 15 O
ct 2024 23:15:33 GMT\r\nServer: scaffolding on HTTPServer2\r\nCache-Control: private\r\nX-XSS-Protection: 0\r\nX-Frame-Options: SAMEORIGIN\r\nAlt-Svc: h3=":443"; ma=2592000,h
3-29=":443"; ma=2592000\r\nAcce')
bytearray(b'" rel="stylesheet"> <meta name="viewport" content="width=device-width, initial-scale=1">  <link href="/static/d05cd6ba/root.css" rel="stylesheet"> </head> <body> 
<span class="filler top"></span>   <div class="logo" title="Google Public DNS"> <div class="logo-text"><span>Public DNS</span></div> </div>  <form action="/query" method="GET
">  <div class="row"> <label class="matter-textfield-outlined"> <input type="text" name="name" placeholder="&nbsp;"> <span>DNS Name</span> <p class="help"> Enter a domain (li
ke example.com) or IP address (like 8.8.8.8 or 2001:4860:4860::8844) here. </p> </label> <button class="matter-button-contained matter-primary" type="submit">Resolve</button>
 </div> </form>  <span class="filler bottom"></span> <footer class="row"> <a href="https://developers.google.com/speed/public-dns">Help</a> <a href="/cache">Cache Flush</a> <
span class="filler"></span> <a href="https://developers.google.com/speed/public-dns/docs/using"> Get Started with Google Public DNS </a> </footer>   <script nonce="m_')
bytearray(b'z8CZg3UI2hfHXAVIjMYw">document.forms[0].name.focus();</script> </body> </html>\r\ncale=1">  <link href="/static/d05cd6ba/root.css" rel="stylesheet"> </head> <body
> <span class="filler top"></span>   <div class="logo" title="Google Public DNS"> <div class="logo-text"><span>Public DNS</span></div> </div>  <form action="/query" method="G
ET">  <div class="row"> <label class="matter-textfield-outlined"> <input type="text" name="name" placeholder="&nbsp;"> <span>DNS Name</span> <p class="help"> Enter a domain (
like example.com) or IP address (like 8.8.8.8 or 2001:4860:4860::8844) here. </p> </label> <button class="matter-button-contained matter-primary" type="submit">Resolve</butto
n> </div> </form>  <span class="filler bottom"></span> <footer class="row"> <a href="https://developers.google.com/speed/public-dns">Help</a> <a href="/cache">Cache Flush</a>
 <span class="filler"></span> <a href="https://developers.google.com/speed/public-dns/docs/using"> Get Started with Google Public DNS </a> </footer>   <script nonce="m_')
bytearray(b'0\r\n\r\n3UI2hfHXAVIjMYw">document.forms[0].name.focus();</script> </body> </html>\r\ncale=1">  <link href="/static/d05cd6ba/root.css" rel="stylesheet"> </head> <
body> <span class="filler top"></span>   <div class="logo" title="Google Public DNS"> <div class="logo-text"><span>Public DNS</span></div> </div>  <form action="/query" metho
d="GET">  <div class="row"> <label class="matter-textfield-outlined"> <input type="text" name="name" placeholder="&nbsp;"> <span>DNS Name</span> <p class="help"> Enter a doma
in (like example.com) or IP address (like 8.8.8.8 or 2001:4860:4860::8844) here. </p> </label> <button class="matter-button-contained matter-primary" type="submit">Resolve</b
utton> </div> </form>  <span class="filler bottom"></span> <footer class="row"> <a href="https://developers.google.com/speed/public-dns">Help</a> <a href="/cache">Cache Flush
</a> <span class="filler"></span> <a href="https://developers.google.com/speed/public-dns/docs/using"> Get Started with Google Public DNS </a> </footer>   <script nonce="m_')

it seems to me like the response isn't finished yet, but it stuck after the last 5 bytes

lealem47 commented 2 weeks ago

Can you share your code that's printing numBytes

RealYukiSan commented 2 weeks ago

import wolfssl
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
secure_sock = wolfssl.wrap_socket(sock)
secure_sock.connect(("dns.google", 443))
secure_sock.write(b"GET / HTTP/1.1\r\nHost: dns.google\r\n\r\n")

response = bytearray(1024)  # A mutable buffer for the recv_into method
data = b""  # To accumulate the received data

while True:
    num_bytes = secure_sock.recv_into(response, 1024)
    print(num_bytes)
    if num_bytes == 0:
        break  # Stop if no more data is received
    data += response[:num_bytes]  # Accumulate the data read into 'data'

secure_sock.close()

wolfSSL / wolfssl-py

Can't read data from the server #57