search

micropython / micropython-lib

Core Python libraries ported to MicroPython
Other
2.4k stars 997 forks source link

Micropython's urequests behaves differently than python's requests #622

Closed gyro-needer closed 1 month ago

gyro-needer commented 1 year ago

Hi,

I am using Micropython's urequests module to access a website but it returns 403 Access Denied error. However, when I use Python's requests module to access the same website, it returns the actual content of the website.

When using urequests:

import urequests

headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

url="https://www.zara.com"

response = urequests.get(url, headers=headers)

print(response.content)

Returns: b'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;www&#46;zara&#46;com&#47;es&#47;es&#47;blazer&#45;traje&#45;estructura&#45;p00706334&#46;html&#63;" on this server.<P>\nReference&#32;&#35;18&#46;c4d31102&#46;1677369242&#46;52175e1f\n</BODY>\n</HTML>\n'


When using Python's requests:

import requests

headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

url="https://www.zara.com"

response = requests.get(url, headers=headers)

print(response.content)

It returns the actual content of the website.

I've tried looking what could cause this problem but haven't been able to make urequests behave like requests.

samveen commented 1 year ago

One debugging method I can suggest is to use an request printer (for example using netcat as nc -l 8080) and send requests to this request printer to examine the difference in the requests sent by the 2 libraries. Also check up on #534

jonnor commented 1 month ago

I ran this code on MicroPython 1.23 on the Unix port. It returns 200 OK with the webpage data. So it seems that this issue has been fixed.

import requests
headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}
url="https://www.zara.com"
response = requests.get(url, headers=headers)
assert response.status_code == 200, response.content
print(response.content)