niklasb / dryscrape

[not actively maintained] A lightweight Python library that uses Webkit to enable easy scraping of dynamic, Javascript-heavy web pages
http://dryscrape.readthedocs.io/
MIT License
533 stars 67 forks source link

Webkit_server error #71

Closed leeadh closed 6 years ago

leeadh commented 7 years ago

Hi I followed the instructions here (http://dryscrape.readthedocs.io/en/latest/) but for some reason Im getting this error

error: [Errno 2] No such file or directory: 'src/webkit_server'

----------------------------------------

Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/var/folders/2t/0xg6g7sd6kq7khygfch7p2980000gn/T/pip-yot635-build/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/2t/0xg6g7sd6kq7khygfch7p2980000gn/T/pip-lk1V2l-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/2t/0xg6g7sd6kq7khygfch7p2980000gn/T/pip-yot635-build/ users-MacBook-Pro:dryscrape user$ cd ..

Any help please?

winnerineast commented 7 years ago

Usually this is not the first and root error, please trace a bit of further back of your logs since I hit the same error but actually it's due to "Undefined symbols for architecture x86_64" which is still my headache now.

yashp1729 commented 7 years ago

I have the same issue and I have tried a few solutions mentioned in similar threads, however, I still get the same issue.

Collecting dryscrape
Requirement already satisfied: xvfbwrapper in /Users/Yash/anaconda/lib/python3.5/site-packages (from dryscrape)
Requirement already satisfied: lxml in /Users/Yash/anaconda/lib/python3.5/site-packages (from dryscrape)
Collecting webkit-server>=1.0 (from dryscrape)
  Using cached webkit-server-1.0.tar.gz
Building wheels for collected packages: webkit-server
  Running setup.py bdist_wheel for webkit-server ... error
  Complete output from command /Users/Yash/anaconda/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/tmp3fnar3ojpip-wheel- --python-tag cp35:
  running bdist_wheel
  running build
  Info: creating stash file /private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/.qmake.stash
  cd src/ && ( test -e Makefile.webkit_server || /Users/Yash/anaconda/bin/qmake /private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/src/webkit_server.pro -spec macx-g++ -o Makefile.webkit_server ) && /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.webkit_server
  Project ERROR: Unknown module(s) in QT: webkitwidgets
  make: *** [sub-src-webkit_server-pro-make_first-ordered] Error 3
  error: [Errno 2] No such file or directory: 'src/webkit_server'

  ----------------------------------------
  Failed building wheel for webkit-server
  Running setup.py clean for webkit-server
Failed to build webkit-server
Installing collected packages: webkit-server, dryscrape
  Running setup.py install for webkit-server ... error
    Complete output from command /Users/Yash/anaconda/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-dw9lvtig-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    cd src/ && ( test -e Makefile.webkit_server || /Users/Yash/anaconda/bin/qmake /private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/src/webkit_server.pro -spec macx-g++ -o Makefile.webkit_server ) && /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.webkit_server
    Project ERROR: Unknown module(s) in QT: webkitwidgets
    make: *** [sub-src-webkit_server-pro-make_first-ordered] Error 3
    error: [Errno 2] No such file or directory: 'src/webkit_server'

    ----------------------------------------
Command "/Users/Yash/anaconda/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-dw9lvtig-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/nw/xkkctrrd1nlfb4zwk0wv8vqc0000gn/T/pip-build-5z0bedbo/webkit-server/
nok commented 7 years ago

Hello,

dryscrape requires Qt@5.5:

brew install qt@5.5

But the Homebrew formula for qt@5.5 is keg only which means binaries like qmake will not be symlinked into your /usr/local/bin directory and therefore will not be available for capybara-webkit.

To force Homebrew to symlink those binaries into your /usr/local/bin directory you can run:

brew link --force qt@5.5

After that you can install dryscrape:

pip install dryscrape
keith555 commented 6 years ago

Hi, I have a similar problem on Ubuntu.

(env) ubuntu@keith-ubuntu-lxc:~/scrapyRarbg$ pip install dryscrape Collecting dryscrape Downloading https://files.pythonhosted.org/packages/b5/75/c45f796ec5bc7f98c38b9ae425390ef5f4a76153c8b5af946adb97e7e622/dryscrape-1.0.tar.gz Collecting webkit_server>=1.0 (from dryscrape) Downloading https://files.pythonhosted.org/packages/29/f2/f4f454cccde75e95359e91fa58f14497350dc97e58534f9003c77eca3dff/webkit-server-1.0.tar.gz (41kB) 100% |████████████████████████████████| 51kB 768kB/s Requirement already satisfied: lxml in /home/ubuntu/anaconda3/lib/python3.6/site-packages (from dryscrape) (4.2.1) Collecting xvfbwrapper (from dryscrape) Downloading https://files.pythonhosted.org/packages/57/b6/4920eabda9b49630dea58745e79f9919aba6408d460afe758bf6e9b21a04/xvfbwrapper-0.2.9.tar.gz Building wheels for collected packages: dryscrape, webkit-server, xvfbwrapper Running setup.py bdist_wheel for dryscrape ... done Stored in directory: /home/ubuntu/.cache/pip/wheels/d5/7e/24/0b5b37166c524082a6fb722bc14c6f885ebb7fcfc7e1563f3e Running setup.py bdist_wheel for webkit-server ... error Complete output from command /home/ubuntu/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-9tr4o1tq/webkit-server/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-5vnintb7 --python-tag cp36: running bdist_wheel running build Info: creating stash file /tmp/pip-install-9tr4o1tq/webkit-server/.qmake.stash cd src/ && ( test -e Makefile.webkit_server || /home/ubuntu/anaconda3/bin/qmake -o Makefile.webkit_server /tmp/pip-install-9tr4o1tq/webkit-server/src/webkit_server.pro ) && make -f Makefile.webkit_server Project ERROR: Unknown module(s) in QT: webkitwidgets Makefile:44: recipe for target 'sub-src-webkit_server-pro-make_first-ordered' failed make: *** [sub-src-webkit_server-pro-make_first-ordered] Error 3 error: [Errno 2] No such file or directory: 'src/webkit_server'


Failed building wheel for webkit-server Running setup.py clean for webkit-server Running setup.py bdist_wheel for xvfbwrapper ... done Stored in directory: /home/ubuntu/.cache/pip/wheels/10/f2/61/cacfaf84b352c223761ea8d19616e3b5ac5c27364da72863f0 Successfully built dryscrape xvfbwrapper Failed to build webkit-server Installing collected packages: webkit-server, xvfbwrapper, dryscrape Running setup.py install for webkit-server ... error Complete output from command /home/ubuntu/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-9tr4o1tq/webkit-server/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-9o92mq48/install-record.txt --single-version-externally-managed --compile: running install running build cd src/ && ( test -e Makefile.webkit_server || /home/ubuntu/anaconda3/bin/qmake -o Makefile.webkit_server /tmp/pip-install-9tr4o1tq/webkit-server/src/webkit_server.pro ) && make -f Makefile.webkit_server Project ERROR: Unknown module(s) in QT: webkitwidgets Makefile:44: recipe for target 'sub-src-webkit_server-pro-make_first-ordered' failed make: *** [sub-src-webkit_server-pro-make_first-ordered] Error 3 error: [Errno 2] No such file or directory: 'src/webkit_server'

----------------------------------------

Command "/home/ubuntu/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-9tr4o1tq/webkit-server/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-9o92mq48/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-9tr4o1tq/webkit-server/

niklasb commented 6 years ago

@keith555 This project is unmaintained, due to QtWebKit being end-of-life. Please use some other Python library for instrumenting QtWebEngine.

keith555 commented 6 years ago

ok. Thanks. I'll just leave this link here for anyone else who comes by. It fixed it for me.