search

bowenpay / wechat-spider

微信公众号爬虫
http://wechatspider.0fenbei.com/
3.16k stars 999 forks source link

mac osx 安装简单教程 #28

Open macoco opened 7 years ago

macoco commented 7 years ago

在mac下成功安装，中间遇到一些小问题，总结一下，帮助后面的小伙伴，我的mac osx 版本10.10.5。 python新人，欢迎大家指导，O(∩_∩)O谢谢~~

1.用virtualenv，安装python2.7

可以用这个命令指定版本:virtualenv -p python2.7 venv 没用过virtualenv可以参考廖老师的教程

2.1安装Mysql-python依赖 yum install python-devel mysql-devel gcc

pip里没有python-devel mysql-devel这2个插件，所以装不上，后面运行也不影响，可以跳过。 gcc装xcode就有了。网上也有不装xcode的方法，但我没试过。

2.2安装lxml依赖 yum install libxslt-devel libxml2-devel

pip里一样没有，可以通过homebrew来安装 我开始也是跳过了，但后面运行程序时报错，在stackoverflow找到这个答案，解决了，运行下面四个命令就行，需要先按照homebrew。

brew install libxml2 brew install libxslt brew link libxml2 --force brew link libxslt --force

2.3安装浏览器环境 selenium依赖.(如果是mac环境，仅需安装firefox， 但确保版本是 firefox 36.0)

Mac firefox 36.0 下载地址：https://ftp.mozilla.org/pub/firefox/releases/36.0.1/mac/

2.4 clone代码 $ git clone https://github.com/bowenpay/wechat-spider.git $ cd wechat-spider $ pip install -r requirements.txt

这一步简单，没有什么问题，pip如果你不是用国内镜像，下载会非常慢，参考这里改成国内镜像。

3) 创建mysql数据库 4) 安装和运行Redis 5) 更新配置文件local_settings

上面三步都简单，照做就行。第六步我出错，报了这个bug： 错误代码 1045 Access denied for user 'root'@'localhost' (using password:YES) 最后原因是我的Mysql设置密码了，在local_settings.py文件的'PASSWORD': '',字段填上密码就行。

后面就一路顺利完成。进去网站发现作者真是太棒了,功能非常完善，里面还有非常好用的搜索功能：

screen shot 2016-11-24 at 1 06 52 am

填完公众号后，最后一步启动四个文件，开始有点懵逼：到底是四个文件依次执行完，还是开四个窗口每个窗口执行一次呢？最后发现是四个窗口都运行才可以。

screen shot 2016-11-22 at 7 35 05 pm

等一下firefox会自动启动，打开搜狗微信，就是成功了。

screen shot 2016-11-22 at 8 04 01 pm

不过现在很大概率会出验证码，需要部署到服务器上自动换ip，等搞定服务器部署我再接着分享。 感谢作者@yijingping 开源这么好的软件。

yijingping commented 7 years ago

多谢 @macoco ， 这个教程很详细，非常👍

mynameissusan commented 7 years ago

2.4 clone代码 $ git clone https://github.com/bowenpay/wechat-spider.git $ cd wechat-spider $ pip install -r requirements.txt

执行 pip install -r requirements.txt 提示“bash：pip：command not found”。 平台：CentOS 7 。步骤1 --> 2.3都没报错。

搜了一圈，在stackoverflow.com也没找到方法。 @macoco 和解？

yijingping commented 7 years ago

那是因为你没有安装pip。执行下面的命令安装：

yum install python-pip
mynameissusan commented 7 years ago

[Labs.MIT@centos wechat-spider]$ sudo yum install python-pip [sudo] password for cornell: Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile

yijingping commented 7 years ago

@mynameissusan 你上面的命令显示 No package python-pip available.，表示你的源里面没有 python-pip 这个库。 你先参考下面的教程先装上pip吧 https://www.liquidweb.com/kb/how-to-install-pip-on-centos-7/ http://pip-cn.readthedocs.io/en/latest/installing.html

另外这里是mac教程的issue，如果还有问题，建一个新的issue吧。

mynameissusan commented 7 years ago

多谢，折腾中 @yijingping

guapier commented 7 years ago

未被限制,可以下载

DEBUG 2017-03-20 12:34:16,089 remote_connection 3995 140735182987264 POST http://127.0.0.1:56142/session {"requiredCapabilities": {}, "desiredCapabilities": {"javascriptEnabled": true, "platform": "ANY", "browserName": "firefox", "version": "", "moz:firefoxOptions": {"profile": "UEsDBBQAAAAIAEhkdEqBD/yDnAMAAMAMAAAHAAAAdXNlci5qc51WTW/cNhC991cUPrVATPijubQn13GAAEFcZGHkSFDUaEUvRTIccuX99xmKq+zaK1Gb3vTBGT7OvPeGEcFz56H546IWQXhw1gdl1sxZreSOpY+rWHUKUVnz3/DxTkpwAeqLd783QiP8+c9v8SiN7VgnXrhsve2Ao/TKBe6j4UF1QDG3V68Deqhqr7b0DkZUGrgRgV45bMEEpIDg45s9grV6owIzEHrrN50wYg2e1QpTgsmQytue3phtGq0MTGN/FluRATPrAh0YGba2/2Tu6dHOZIaXAAaHxdFRvYDlc8zUZwTy8PT5jt0ycSjmaWrh3Fk5x3oE0NBB8Dvm4RnkXNoRAoLwst3vUEaLooHhOXGjE7onqpQx7Qt9SWdAJrS2Pa92vIZGRB0mYb0mYAtChza/E1K/VRJYozyGr9FMb3lgkkCMRL5oaBekMnDibwRfrgbtHqJjLdHWEZ+4pUxe1cA6DHagzIVaG+vhopwgBdPiq0Lfx9RFQEFUyKjQ5tE8Opg586GXg0IJKTXGw3D8hhTIpRfYTseeMtfYoJrdEw6lmohAkNGrsGMS3b79ZVjJDXplatuzOnbuiDJFIckW5Obedo6soFI67WjUug16t1CFNz1MPROVjeHvSguzmWmcpqN7cqgyoUdjW3a0Eyb1oGUyw+h1GVEN26RlZITHepld59yiabteJ+WUlpegMVHXKrmeWEC5t13WhkBsbxW2u8v025HsLjWYdUiEu3n/ftZdSxAdsdYFTPTnna2FXmjLvGuMBdnX8eEcW35ldMWNX5wiuZGzMSRbECGS6kh536Pyi5u0QGEDyT9kQ/w3/5ln3n62cUtOwGsSoVlzbcXZA+EX50duwrD+Zk7ZvUkA2Dh906T81oJZJYoRvAVjUOi02EH9cH21+kK2I5OS/ppebKAnNiS+/m8ujBOkGP/T3hqlgZ7IT7ySgVuviEg834nK5/Ig6nQ8CMP8/ZgG1h3hkcu3h0pbudEKQxnkqL5c98txzhIVQ8Rfrkt0qYlFaRzdQkQMdqZ0aTQaOn4HXQV+NbzOwPnpu8ps2EBoanCeEqd8OypQ2v1D1sFKUlxKfz0zZjNl2LLjkrZyKThJmT8jh5d0I1MZ/WnDDrLvW0WCSf0qq/7oWjJc9o6uJRL8zBV34hJwry3OjNsz7u5nud+hI6nsPk+CmwWRvFXH7XRu+lQ/mm9Dm/H6iimqgn36+vm8GY2gG4wuczY3Na38AVBLAQIUAxQAAAAIAEhkdEqBD/yDnAMAAMAMAAAHAAAAAAAAAAAAAACkgQAAAAB1c2VyLmpzUEsFBgAAAAABAAEANQAAAMEDAAAAAA=="}, "marionette": true}} DEBUG 2017-03-20 12:34:17,122 remote_connection 3995 140735182987264 Finished Request 'status' Traceback (most recent call last): File "downloader.py", line 97, in downloader.run() File "downloader.py", line 78, in run with SeleniumDownloaderBackend(proxy=proxy) as browser: File "/Users/macbook/Downloads/wechat-spider-master/wechat/downloaders.py", line 42, in enter self.browser = self.get_browser(self.proxy) File "/Users/macbook/Downloads/wechat-spider-master/wechat/downloaders.py", line 90, in get_browser browser = webdriver.Firefox(firefox_profile=firefox_profile) File "/Library/Python/2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 150, in init keep_alive=True) File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 92, in init self.start_session(desired_capabilities, browser_profile) File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 179, in start_session response = self.execute(Command.NEW_SESSION, capabilities) File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute self.error_handler.check_response(response) File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 104, in check_response status = value["status"] KeyError: 'status' mac 运行download.py 出现这个问题，不知道你碰到过没

yijingping commented 7 years ago

@guapier 注意mac下，firefox版本必须是36.0 才不会出问题。

200684632 commented 7 years ago

@macoco @yijingping 为什么我这边那个36版本的firefox下载了用不了 本人使用的是mac 10.12.3的版本。所以想请问一下是不是系统的问题。

northleafup commented 7 years ago

@200684632 您好，请问您后来是如何解决这个浏览器的问题的？

200684632 commented 7 years ago

@northleafup 这个问题我没有解决。我试了很多次 ，应该是os的版本问题，所以我帮不了你。对不住。 如果你后边解决了，麻烦告知我一声，🙏。

northleafup commented 7 years ago

@200684632 您好，请参考http://xclient.info/a/74559ea2-7870-b992-ed53-52a9d988e382.html 这个页面的方式安装firefox 36，由于firefox 总是自动升级，建议按以下方式设置防止其自动升级：将/Users/用户名/Library/Caches/Mozilla//updates/Applications/Firefox/ 路径下的文件改为root权限，同时将它们改为无权限(sudo chmod 000 *)，这样Firefox就不能再升级了

wcmbeta commented 6 years ago

macOS 10.12.6,firefox 36.0.1,python 2.7.10,其他都一样,但是打不开主页localhost:8001,没学过Python😂

Request Method: GET

http://localhost:8001/ 1.8.1 ImportError No module named bs4 ....../wechat-spider/wechat/extractors.py in , line 8 /usr/bin/python 2.7.10 ['....../Desktop/wechat-spider/wechat-spider', '/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Python/2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC'] 星期三, 22 十一月 2017 17:27:38 +0800

Kingson commented 6 years ago

No module named bs4 要装美丽的汤

wcmbeta commented 6 years ago

@Kingson 原来是批量安装依赖的时候没成功，pip list 看到有个3.2.1的旧版本， sudo pip install -U beautifulsoup4 单独更新就成功了

tom510230 commented 6 years ago

补充一下，在安装mysql-python的时候报错了

Requirement already satisfied: six>=1.5 in /usr/local/lib/python2.7/site-packages (from python-dateutil==2.5.0->-r requirements.txt (line 10))
Building wheels for collected packages: MySQL-python
  Running setup.py bdist_wheel for MySQL-python ... error
  Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/ky/syfrlzpn2bb8fmn5pr51tstm0000gp/T/pip-build-sHJvzD/MySQL-python/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/ky/syfrlzpn2bb8fmn5pr51tstm0000gp/T/tmptbFzqdpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.13-x86_64-2.7
  copying _mysql_exceptions.py -> build/lib.macosx-10.13-x86_64-2.7
  creating build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  copying MySQLdb/__init__.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  copying MySQLdb/converters.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  copying MySQLdb/connections.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  copying MySQLdb/cursors.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  copying MySQLdb/release.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  copying MySQLdb/times.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb
  creating build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/__init__.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/CR.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/ER.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/FLAG.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/REFRESH.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  copying MySQLdb/constants/CLIENT.py -> build/lib.macosx-10.13-x86_64-2.7/MySQLdb/constants
  running build_ext
  building '_mysql' extension
  creating build/temp.macosx-10.13-x86_64-2.7
  clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/opt/openssl/include -Dversion_info=(1,2,5,'final',1) -D__version__=1.2.5 -I/Applications/XAMPP/xamppfiles/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _mysql.c -o build/temp.macosx-10.13-x86_64-2.7/_mysql.o -arch x86_64 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing
  clang: error: unknown argument: '-fabi-version=2'
  error: command 'clang' failed with exit status 1

死活认为clang编译不过，经过搜索，找到了这篇文章，用这个命令解决了

brew install mysql
pip install MySQL-python

最后安装的时候可能还会报错 OSError: [Errno 13] Permission denied: '/usr/local/selenium' 改用sudo pip install -r requirements.txt就好