binux / pyspider

A Powerful Spider(Web Crawler) System in Python.
http://docs.pyspider.org/
Apache License 2.0
16.46k stars 3.69k forks source link

can not support python3.7 #817

Open ROSClub opened 6 years ago

ROSClub commented 6 years ago

when i run pyspider . error message: Traceback (most recent call last): File "/usr/local/bin/pyspider", line 7, in from pyspider.run import main File "/usr/local/lib/python3.7/site-packages/pyspider/run.py", line 231 async=True, get_object=False, no_input=False):

mac os 10.13 python3.7

aaaaasam commented 6 years ago

Change all async to other, change async to aasync here.

sam@mac:/usr/local/lib/python3.7/site-packages/pyspider$ grep -Ri async ./* | grep -v async
Binary file ./__pycache__/run.cpython-37.pyc matches
./fetcher/tornado_fetcher.py:from tornado.curl_httpclient import CurlAsyncHTTPClient
./fetcher/tornado_fetcher.py:from tornado.simple_httpclient import SimpleAsyncHTTPClient
./fetcher/tornado_fetcher.py:class MyCurlAsyncHTTPClient(CurlAsyncHTTPClient):
./fetcher/tornado_fetcher.py:class MySimpleAsyncHTTPClient(SimpleAsyncHTTPClient):
./fetcher/tornado_fetcher.py:            self.http_client = MyCurlAsyncHTTPClient(max_clients=self.poolsize,
./fetcher/tornado_fetcher.py:            self.http_client = tornado.httpclient.HTTPClient(MyCurlAsyncHTTPClient, max_clients=self.poolsize)
Binary file ./fetcher/__pycache__/tornado_fetcher.cpython-37.pyc matches
Binary file ./webui/__pycache__/app.cpython-37.pyc matches
sam@mac:/usr/local/lib/python3.7/site-packages/pyspider$ pwd
/usr/local/lib/python3.7/site-packages/pyspider
aaaaasam commented 6 years ago

What’s New In Python 3.7

Backwards incompatible syntax changes:

async and await are now reserved keywords.

look here

ShawnXiee commented 6 years ago

Have you solved this problem?

sdvcrx commented 6 years ago

803 already fixed, just need release a new version.

bigmangos commented 6 years ago

@binux Thanks for releasing a new version to solve this bug.

lymanland commented 5 years ago

C:\Users\dell-pc>pyspider C:\Users\dell-pc\AppData\Roaming\Python\Python37\site-packages\pyspider\libs\uti ls.py:196: FutureWarning: timeout is not supported on your platform. warnings.warn("timeout is not supported on your platform.", FutureWarning) phantomjs fetcher running on port 25555 [I 181017 20:29:08 run:420] phantomjs exited. Exception in thread Thread-4: Traceback (most recent call last): File "c:\programdata\anaconda3\lib\threading.py", line 917, in _bootstrap_inne r self.run() File "c:\programdata\anaconda3\lib\threading.py", line 865, in run self._target(*self._args, self._kwargs) File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 535, in invoke return callback(*args, *kwargs) File "c:\programdata\anaconda3\lib\site-packages\click\decorators.py", line 17 , in new_func return f(get_current_context(), args, kwargs) File "C:\Users\dell-pc\AppData\Roaming\Python\Python37\site-packages\pyspider\ run.py", line 236, in fetcher Fetcher = load_cls(None, None, fetcher_cls) File "C:\Users\dell-pc\AppData\Roaming\Python\Python37\site-packages\pyspider\ run.py", line 48, in load_cls return utils.load_object(value) File "C:\Users\dell-pc\AppData\Roaming\Python\Python37\site-packages\pyspider\ libs\utils.py", line 369, in load_object module = import(module_name, globals(), locals(), [object_name]) File "C:\Users\dell-pc\AppData\Roaming\Python\Python37\site-packages\pyspider\ fetcher__init__.py", line 1, in from .tornado_fetcher import Fetcher File "C:\Users\dell-pc\AppData\Roaming\Python\Python37\site-packages\pyspider\ fetcher\tornado_fetcher.py", line 89 self.async = async_mode ^ SyntaxError: invalid syntax

[I 181017 20:29:08 result_worker:49] result_worker starting... [I 181017 20:29:12 processor:211] processor starting... [I 181017 20:29:12 scheduler:675] scheduler starting... [I 181017 20:29:12 scheduler:614] in 5m: new:0,success:0,retry:0,failed:0 [I 181017 20:29:13 scheduler:810] scheduler.xmlrpc listening on 127.0.0.1:23333 [I 181017 20:30:12 scheduler:614] in 5m: new:0,success:0,retry:0,failed:0 [I 181017 20:31:13 scheduler:614] in 5m: new:0,success:0,retry:0,failed:0 [I 181017 20:32:13 scheduler:614] in 5m: new:0,success:0,retry:0,failed:0

我是安装Anaconda3后,再pip install pyspider ,也是遇到这个报错 更新了最新版本,运行pyspider报错,不能打开

有谁又遇到么?

Harold-the-Axeman commented 5 years ago

Mac os 使用 pip 安装的,python3.7现在还有这个问题

44886 commented 5 years ago

修改 C:\Users\你的用户名\AppData\Local\Programs\Python\Python37-32\Lib\site-packages\pyspider\run.py 中的async为其他字符就可以啦

wangzhihaiSF commented 5 years ago

如果是用 Anaconda3 ,可以考虑将python 降级为 3.6 https://www.jianshu.com/p/618e5ebafb01

sadegh-khan commented 4 years ago

@binux Thanks for releasing a new version to solve this bug.

i already have this bug !!! what version do you mean?

gitfourteen commented 4 years ago

1 year and a half have passed and this issue is still hovering... macOS Catalina python 3.7.5 pipenv

ludwigwittgenstein2 commented 4 years ago

I solved it.

Change to async_ in all places Use vim.

%s/async/async/g (Replace all async to async)

YongjiangChen commented 1 year ago

%s/async/async_mode/g