search

psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.19k stars 9.33k forks source link

Same code works with Python 3.10 but not with 3.11 #6768

Closed djtech-dev closed 4 months ago

djtech-dev commented 4 months ago

The investpy library doesn't work under CPython 3.11 while it works without issues with CPython 3.10

Expected Result

In the search.py file, at line 157 (function search_quotes), I get a ConnectionError (not raised by requests, but connected to the status_code returned by requests). This error is weirdly only raised on Python 3.11, while Python 3.10 works without issue. I've edited the code in order to print some important informations (the parameters and the output for the requests's POST function).

Snippet (original, can be found at the link before):

req = requests.post(url, headers=headers, data=params)

if req.status_code != 200:
    raise ConnectionError(
        f"ERR#0015: error {req.status_code}, try again later."
    )

data = req.json()

Snippet (modified, added print statements and a static user agent, while the original project generated each time a new user agent in order to fake activity):

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 GTB5', 'X-Requested-With': 'XMLHttpRequest', 'Accept': 'text/html', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive'}
print(params)
print("---")
print(headers)
req = requests.post(url, headers=headers, data=params)
print("---")
print(req)

if req.status_code != 200:
  raise ConnectionError(
      f"ERR#0015: error {req.status_code}, try again later."
  )

data = req.json()

Result on CPython 3.10:

>>> investsearch.search_quotes("IBM")
{'search_text': 'IBM', 'tab': 'quotes', 'isFilter': True, 'limit': 270, 'offset': 0}
---
{'User-Agent': 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 GTB5', 'X-Requested-With': 'XMLHttpRequest', 'Accept': 'text/html', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive'}
---
<Response [200]>
[<investpy.utils.search_obj.SearchObj object at 0x7fdd838952d0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd837e6680>, <investpy.utils.search_obj.SearchObj object at 0x7fdd8381d900>, <investpy.utils.search_obj.SearchObj object at 0x7fdd838cf370>, <investpy.utils.search_obj.SearchObj object at 0x7fdd838cc8e0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd838cf340>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb190>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb1f0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb250>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb280>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb2b0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb160>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9cc0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9cf0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9d20>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfbe20>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9c90>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9ff0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb220>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfb2e0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfa020>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfa050>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9ed0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9f60>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfa0b0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfa200>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bfbf70>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9f90>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9f30>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9ea0>, <investpy.utils.search_obj.SearchObj object at 0x7fdd74bf9e70>]

Actual Result

Results on CPython 3.11:

>>> investsearch.search_quotes("IBM")
{'search_text': 'IBM', 'tab': 'quotes', 'isFilter': True, 'limit': 270, 'offset': 0}
---
{'User-Agent': 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 GTB5', 'X-Requested-With': 'XMLHttpRequest', 'Accept': 'text/html', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive'}
---
<Response [403]>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/giuseppe/project_348/Financials-Extension/investpy/investpy/search.py", line 166, in search_quotes
    raise ConnectionError(
ConnectionError: ERR#0015: error 403, try again later.

Reproduction Steps

Use the snippet/library showed before.

System Information

Running under CPython 3.10:

{
  "chardet": {
    "version": "5.2.0"
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.14"
  },
  "platform": {
    "release": "6.8.12-amd64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "30200010"
  },
  "urllib3": {
    "version": "2.2.1"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}

Running under CPython 3.11:

/home/giuseppe/.local/lib/python3.11/site-packages/requests/help.py:24: DeprecationWarning: 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in a future release of urllib3 2.x. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680
  from urllib3.contrib import pyopenssl
{
  "chardet": {
    "version": "5.2.0"
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": "42.0.5"
  },
  "idna": {
    "version": "3.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.9"
  },
  "platform": {
    "release": "6.8.12-amd64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "30200010",
    "version": "24.1.0"
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30200020"
  },
  "urllib3": {
    "version": "1.26.18"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}
Mariatta commented 4 months ago

403 error is issue with authentication/permission, so I don't think it is related to Python version or the requests library. Perhaps you can ask questions on Stack Overflow on why you're getting 403 error

nateprewitt commented 4 months ago

As @Mariatta correctly noted, the response behavior from the service isn't something we control. There are a number of reasons you may be receiving a different status code depending on Python version, both from your environment and potential service-side decisions based on client fingerprinting.

StackOverflow is the best place to ask these questions unless you're able to provide a reproducible example without the service in question.