Scrape tutorial - lbxml2 related errors installing Scrapy and unresolved external errors installing psycopg2

[mlukjanska]

Systeminfo: MS Windows 7, Python 2.7

Running a command pip install -r requirements.txt from within virtualenv I kept getting bunch of errors, once running the requirements one by one it turned out that issue was with scrapy and psycopg2.

Solving issue with Scrapy

The installation would fail with the following error

    c:\users\<user>\appdata\local\temp\xmlXPathInitjfnyng.c(1) : fatal error C1083: Cannot open include file: 'libxml/xpat
h.h': No such file or directory
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'C:\\Users\\<user>\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\c
l.exe' failed with exit status 2

    ----------------------------------------
Command "c:\users\<user>\.virtualenvs\scrapeproj\scripts\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users
\\<user>\\appdata\\local\\temp\\pip-build-8nda_z\\lxml\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).r
ead().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\<user>\appdata\local\temp\pip-nkczkk-record\ins
tall-record.txt --single-version-externally-managed --compile --install-headers c:\users\<user>\.virtualenvs\scrapeproj\in
clude\site\python2.7\lxml" failed with error code 1 in c:\users\<user>\appdata\local\temp\pip-build-8nda_z\lxml

Simply running pip install lxml within virtualenv did not help (suggestion from here),

but running easy_install.exe lxml as indicated here helped

Solving issue with psycopg2

While installing the psycopg2==2.5.4 I kept getting the following errors:

build\lib.win32-2.7\psycopg2\_psycopg.pyd : fatal error LNK1120: 62 unresolved externals
error: Setup script exited with error: command 'C:\\Users\\mita\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++
 for Python\\9.0\\VC\\Bin\\link.exe' failed with exit status 1120

and managed to solve them by installing the binaries (see more details here) by running pip install git+https://github.com/nwcell/psycopg2-windows.git@win32-py27#egg=psycopg2 within virtualenv

[spizeck]

I had the error in Mint 17.3. Unable to install Scrapy...

Command "/home/chad/Envs/ScrapeProf/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-ZrTWdn/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-0AOCRj-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/chad/Envs/ScrapeProf/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-build-ZrTWdn/cffi

[eenblam]

@mlukjanska libxml2 is a C library. You probably needed the development versions of libxml2 and libxslt. I guess it's good that easy_install took care of that for you.

@spizeck Is that the entire error message? What command did you enter? If you're having the same problems that @mlukjanska is having, then you would resolve it with

sudo apt-get update
sudo apt-get install libxml2-dev libxslt1-dev

However, you seem to be having a separate issue. Issue #170 may be more relevant.

[bhattraideb]

I'm still getting error by easy_install in win7.
could not find function xmlCheckVersion in library libXml2. Is libxml2 installed?

[bhattraideb]

I'm still getting error by easy_install in win7.
could not find function xmlCheckVersion in library libXml2. Is libxml2 installed?

[bhattraideb]

"pip install lxml==3.4.2" command helped me to solve this issue.