spyoungtech / grequests

Requests + Gevent = <3
https://pypi.python.org/pypi/grequests
BSD 2-Clause "Simplified" License
4.48k stars 331 forks source link

Python 3.6 the fastest environment? #168

Open ElizarovEugene opened 1 year ago

ElizarovEugene commented 1 year ago

Hi I'm a bit new to python and this is my first time working with your library. After migrating one of my scripts using grequests from an old server with python 3.6 to a new server with python 3.11, I noticed that it began to be executed several times old server: centos 7, python 3.6 - it's a VM - 2vPCU, 8Gb $ python parse.py --- 11.287319898605347 seconds --- new server: ubuntu 22.04, python 3.11 - it's a VM - 2vPCU, 8Gb $ python3 parse.py --- 71.94743847846985 seconds ---

I started looking for what exactly is the problem. I already took the Debian 11, installed Anaconda there with different versions of python and ran a test with my own script (I ran each test twice, to be sure). At the same time I tried 3.8

(py_env3.6) korp@debian:~$ python parse.py
--- 11.409996509552002 seconds ---
(py_env3.6) korp@debian:~$ python parse.py
--- 12.19371223449707 seconds ---
(py_env3.8) korp@debian:~$ python parse.py
--- 33.36833906173706 seconds ---
(py_env3.8) korp@debian:~$ python parse.py
--- 34.453060150146484 seconds ---
(py_env3.11) korp@debian:~$ python parse.py
--- 33.38819909095764 seconds ---
(py_env3.11) korp@debian:~$ python parse.py
--- 33.33392095565796 seconds ---

Is this expected behavior or am I doing something wrong?

spyoungtech commented 1 year ago

Hmmm. I'm not sure why Python version alone would cause this. I would maybe suspect the version of gevent and/or requests being used as a possible cause. It might also be helpful if this could be reproduced with a minimal example (say, using httpbin.org and some minimal code that is compatible between 3.6 and 3.11) and I could investigate more closely.

ElizarovEugene commented 1 year ago

https://www.dropbox.com/s/at1yvchyu9k4ted/parse.zip?dl=0 my script + db