tools/install.sh fails on install of matplotlib from requirements.txt on Digital Ocean droplet

piratica commented 8 years ago

Issue Description

Attempting to setup a Digital Ocean droplet to test a site. Tried via PTF and then directly but failed each time at matplotlib. This seems to be specific to Digital Ocean droplets but seems to be distro independent.

Reproduction Steps

Initially used PTF for the install but it failed. Went directly into /pentest/exploitation/king-phisher and attempted to run tools/install.sh but failed again at the same spot. Noted that pip had been installing matplotlib and had gotten to 99% and then bombed (I was too lazy to take them down but it was consistent across Ubuntu, Debian and Fedora droplets).

Work Around so far

install matplotlib via apt
- sudo apt-get install python-mpltoolkits.basemap
- sudo apt-get install python-mpltoolkits.basemap-data
clean up apt (not sure why this is required yet but it seems to be)
- sudo apt-get update --fix-missing
comment the matplotlib in requirements.txt
- sed -i 's/^matplotlib/#matplotlib/'
tools/install.sh
go phishing :)

zeroSteiner commented 8 years ago

For Digital Ocean, the $20/month droplet needs to be used to have the 2GB of RAM in order to compile matplotlib. I ran into this error myself two weeks ago. You can temporarily either:

Upgrade to the 2GB level then downgrade when you're done
Mount disk space as extra swap space

When going through the install you'll notice an out of memory error as it's trying to decode some data.

piratica commented 8 years ago

Very cool, thank you for the quick response on this. I'll give that a shot and follow-up here but it seems like this is a non-issue.

piratica commented 8 years ago

Ok, just spun up one of the $20 per month droplets and did the following:

created droplet
logged in via SSH
apt-get install git
git clone https://github.com/trustedsec/ptf.git
(inside ptf) >use modules/explotation/kingphisher
(inside ptf) >install
(inside ptf) installer seemed to go well but bombed again at matplotlib `Collecting matplotlib>=1.5.0 (from -r requirements.txt (line 13)) Downloading matplotlib-1.5.1.tar.gz (54.0MB) 100% |████████████████████████████████| 54.0MB 13kB/s Complete output from command python setup.py egg_info: IMPORTANT WARNING: pkg-config is not installed. matplotlib may not be able to find some of its dependencies

Edit setup.cfg to change the build options

BUILDING MATPLOTLIB matplotlib: yes [1.5.1] python: yes [2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2]] platform: yes [linux2]

REQUIRED DEPENDENCIES AND EXTENSIONS numpy: yes [not found. pip may install it below.] dateutil: yes [dateutil was not found. It is required for date axis support. pip/easy_install may attempt to install it after matplotlib.] pytz: yes [pytz was not found. pip will attempt to install it after matplotlib.] cycler: yes [cycler was not found. pip will attempt to install it after matplotlib.] tornado: yes [tornado was not found. It is required for the WebAgg backend. pip/easy_install may attempt to install it after matplotlib.] pyparsing: yes [pyparsing was not found. It is required for mathtext support. pip/easy_install may attempt to install it after matplotlib.] libagg: yes [pkg-config information for 'libagg' could not be found. Using local copy.] freetype: no [The C/C++ header for freetype2 (ft2build.h) could not be found. You may need to install the development package.] png: yes [version 1.2.50] qhull: yes [pkg-config information for 'qhull' could not be found. Using local copy.]

OPTIONAL SUBPACKAGES sample_data: yes [installing] toolkits: yes [installing] tests: yes [nose 0.11.1 or later is required to run the matplotlib test suite. Please install it with pip or your preferred tool to run the test suite / mock is required to run the matplotlib test suite. Please install it with pip or your preferred tool to run the test suite] toolkits_tests: yes [nose 0.11.1 or later is required to run the matplotlib test suite. Please install it with pip or your preferred tool to run the test suite / mock is required to run the matplotlib test suite. Please install it with pip or your preferred tool to run the test suite]

OPTIONAL BACKEND EXTENSIONS macosx: no [Mac OS-X only] qt5agg: no [PyQt5 not found] qt4agg: no [PySide not found; PyQt4 not found] gtk3agg: no [Requires pygobject to be installed.] gtk3cairo: no [Requires cairocffi or pycairo to be installed.] gtkagg: no [Requires pygtk] tkagg: no [TKAgg requires Tkinter.] wxagg: no [requires wxPython] gtk: no [Requires pygtk] agg: yes [installing] cairo: no [cairocffi or pycairo not found] windowing: no [Microsoft Windows only]

OPTIONAL LATEX DEPENDENCIES dvipng: no ghostscript: no latex: no pdftops: no

OPTIONAL PACKAGE DATA dlls: no [skipping due to configuration]

============================================================================ * The following required packages can not be built: * freetype

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-GAQYMq/matplotlib/ Failed to install python requirements with pip `
exited ptf and went to /pentest/exploitation/kingphisher and ran tools/install.sh and got the same error while watching /var/log/syslog and htop but didn't see any log entries regarding memory and htop didn't seem to indicate that we were [ab]using ram.
will continue poking around to see what I can find.

zeroSteiner commented 8 years ago

It's complaining that freetype can't be built. If you're on Ubuntu, I think I remember there being an error with the freetype header not being in the correct location. Try running sudo ln -s /usr/include/freetype2/ft2build.h /usr/include/ft2build.h first.

piratica commented 8 years ago

Ok, so down the rabbit hole.

First, I feel like a dork for not catching the complaint about freetype (the message "The following required packages can not be built: * freetype" should have been a hint).
Second, I created the symlink and re-ran the tools/install.sh and this time matplotlib installed but I got an error when it was installing psycopg2 that it needed postgresql-server-dev-X.Y.
The output is somewhat verbose so I'm going to go through it to see what I may have missed previously and will post back with my findings. On the Backbox install that worked, I don't recall if we had postgresql-server-dev. but I've already deleted the VM (it worked beautifully though :) ). Additionally, the initial testing of the one that we did earlier with my work around seems to have worked (we sent 2 emails to the staff that will be running the test, so I don't know that it's a reliable sample).
I appreciate your prompt feedback on this and don't want you to think that there's any urgency on this. I'm posting the info here a) in case someone else has the same issue and b) because I'm too lazy to fire up TextWrangler or gedit to keep my notes locally.

zeroSteiner commented 8 years ago

@wolfthefallen is working on a work around for the freetype issue. The necessary postgresql-server-dev packages should be installed automatically though. What flavor of Linux are you getting that error on?

piratica commented 8 years ago

This one is an Ubuntu droplet at Digital Ocean, a completely vanilla install. It's a burner so, if you want to poke around on it let me know. We'll need to find a way to pass credentials (public key?) but I'm happy to let you log on and poke around if it would help.

zeroSteiner commented 8 years ago

@piratica we were able to reproduce the fix the freetype build issue. Turns out that running apt-get update prior to installing any packages results in the freetype header being placed in the location where the install script expects it to be. This was patched in #80 which has been pushed up to the dev branch and will make its way to master in the next release.

Are there any outstanding issues left? I believe we've figured out both the memory requirements and addressed the freetype build error now.

piratica commented 8 years ago

Just tried again (git clone from the dev branch on this one) on a fresh droplet and got the same freetype error again. It seems to work everywhere except Digital Ocean and I've got a workaround that seems to be working well.
Also, apologies on the delayed response on this.

wolfthefallen commented 8 years ago

@piratica Thank you for trying out the fix in the dev branch. I'm having issues reproducing it with the new change, If you can please put the out put of the install script into a file or paste bin so I can review it that would be a great help. Also please let me know what Digital Ocean (DO) image and size you are using.

tools/install.sh > install_output.txt

I have a feeling the libfreetype is failing to install via apt-get, and since the install.sh script is not set to exit on failure their it is making it to pip install and failing there as the package is not in the correct place.

piratica commented 8 years ago

Resetting and trying again. Below is a play by play. The droplet is one of the $5 per month droplets with Ubuntu 14.04 64bit that's been resized to include 2GB RAM. This is a burner so all commands are running as root (yes, bad plan but the box has a maximum lifespan of about 24 hours).

Clean droplet
apt-get install fail2ban unzip
apt-get update
apt-get -update --fix-missing (seems to be a reflex, I had typed it before thinking about it)
mkdir -p /pentest/exploitation/
cd /pentest/exploitation/
wget -c https://github.com/securestate/king-phisher/archive/dev.zip
unzip dev.zip
cd king-phisher-dev/
tools/install.sh > install_output.txt
things seemed to stop (not sure why yet) but the following was the last thing in install_output.txt. I let it sit for 17 minutes and got impatient and killed it. I'm about to rebuild the VM, wash, rinse and repeat and be a little more patient this time but wanted to get you my process in case you spotted something dumb that I was doing.

Downloading MarkupSafe-0.23.tar.gz Collecting matplotlib>=1.5.0 (from -r requirements.txt (line 13)) Downloading matplotlib-1.5.1.tar.gz (54.0MB)

piratica commented 8 years ago

Ok, repeated above noted process, walked away (as to prevent any unwelcome tinkering) and, when I returned, I saw the following:

Starting the King Phisher service
king-phisher start/running, process 26053

I've attached the install_output.txt in case there's any value there but this time it seems to have worked like a charm. I'll poke around with it a bit and let you know if I am able to break anything else :) install_output.txt

zeroSteiner commented 8 years ago

That's very odd, I do know the install script can be picky when run from another branch. I did see the message stating it was updating the apt-get cache implying that it had @wolfthefallen's changes. Very odd that it didn't work the previous time. We haven't made any changes since then.

zeroSteiner commented 8 years ago

We released version 1.2.0 last Friday so those changes to the install script have been merged into master. @piratica are you still experiencing issues or can this ticket be closed?

zeroSteiner commented 8 years ago

I'm going to close this ticket out as I'm not aware of any additional issues that need to be addressed at this time. If you think of something else that's related you can comment on this thread or else open a new ticket. Thanks!

rsmusllp / king-phisher