scrapy.cmdline not found

[treise]

So I installed python 2.7.13, ran the pip install -r requirements.txt and then on my first test of xsscrapy I get the following error:

root@debian:/opt/xsscrapy# ./xsscrapy.py -u http://192.168.1.12/mutillidae/index.php?page=dns-lookup.php Traceback (most recent call last): File "./xsscrapy.py", line 4, in from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline

This makes little sense to me considering scrapy is in my path.

root@debian:/opt/xsscrapy# echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

root@debian:/opt/xsscrapy# whereis scrapy scrapy: /usr/local/bin/scrapy

It is weird, this works flawless on kali 2.0 which is based on debian. I'm doing this install on debian 8 and its broken... dunno how that works but the world is a strange place.

[triploopnet]

I'm also having this issue. Anyone have a fix?

$ python xsscrapy.py --help
Traceback (most recent call last):
  File "xsscrapy.py", line 4, in <module>
    from scrapy.cmdline import execute
ImportError: No module named scrapy.cmdline

[decidedlygray]

Are you sure Scrapy and other dependencies were installed correctly when you ran pip install -r requirements.txt?

On a fresh Debian machine, I used the following commands to get set up and it works as expected:

# Initial update of fresh debian system
apt update && apt upgrade -y && apt install git python python-pip -y

# Not required - set up new isolated Python2 environment for xsscrapy
cd /opt
pip install virtualenv
virtualenv venv_xsscrapy
source venv_xsscrapy/bin/activate

# Clone the repository and attempt to install requirements
git clone https://github.com/DanMcInerney/xsscrapy.git
cd xsscrapy
pip install -r requirements.txt

The above actually fails on a fresh system. I needed to install python-dev, then everything worked as expected:

apt install python-dev -y
pip install --no-cache-dir -r requirements.txt

Then it works as expected:

(venv_xsscrapy) root@antlet26:/opt/xsscrapy# python xsscrapy.py --help
usage: xsscrapy.py [-h] [-u URL] [-l LOGIN] [-p PASSWORD] [-c CONNECTIONS]
                   [-r RATELIMIT] [--basic] [-k COOKIE]

optional arguments:
  -h, --help            show this help message and exit
  -u URL, --url URL     URL to scan; -u http://example.com
  -l LOGIN, --login LOGIN
                        Login name; -l danmcinerney
  -p PASSWORD, --password PASSWORD
                        Password; -p pa$$w0rd
  -c CONNECTIONS, --connections CONNECTIONS
                        Set the max number of simultaneous connections
                        allowed, default=30
  -r RATELIMIT, --ratelimit RATELIMIT
                        Rate in requests per minute, default=0
  --basic               Use HTTP Basic Auth to login
  -k COOKIE, --cookie COOKIE
                        Cookie key; --cookie
                        SessionID=afgh3193e9103bca9318031bcdf
(venv_xsscrapy) root@antlet26:/opt/xsscrapy#

For next steps debugging, please provide the output of this command:

pip install -r requirements.txt | grep "Scrapy"

[DanMcInerney]

yeah sounds like problem with your scrapy install.

[BIGSergio123]

helped me so much , thanks

[shad02]

After doing above fix....i am getting this problem..can you solve this problem