Installer script?

[dufywihi]

Maybe we can work on an installer script to expedite the process and install chomper with a single ($ ./installer.sh) command? How's your shell scripting? I added comments, let me know what you think.

#!/bin/bash
    # make sure installer script isnt run as root
    if [ "$(id -u)" = "0" ]; then
    echo -e "\nThis script must not be run as root\n" 1>&2;
    exit 1;
    # debian usage warning
    if [[ $(lsb_release -is) != 'Debian' ]]; then
        echo -e "\nThis script was designed for Debian. Press [Enter] to proceed. Otherwise, press Ctrl+c to exit installer.\n";
        read;
    fi;
    fi;
    # update apt index
    sudo apt-get update;

    # make sure git is installed
    if ! [ -x "$(command -v git)" ]; then
    echo -e "\ngit not found. Installing...\n";
    sudo apt-get install git -y;
    fi;
    # check for chomper dir, in case script is run multiple times
    if [ -d ~/chomper ]; then
    echo -e "\nChomper directory detected. Remove the directory with '$ rm -rf ~/chomper' to use this script.\n";
    exit 1;
    else
    cd ~/ && git clone https://github.com/aniketpanjwani/chomper.git;
    cd chomper;
    fi;
    # shouldnt `export` and `source` come before the `make` command? you have it reversed in the README.md
    export PATH="/home/$USER/chomper/bin:$PATH" >> ~/.bashrc;
    source /home/$USER/.bashrc;
    make init

[aniketpanjwani]

This is awesome! I'm definitely amenable to having a shell installer script - I really want to make installation as easy as possible. I wanted to do something similar to this, but my shell scripting is just okay, so I was procrastinating a bit on writing it. However, this looks very well written, so I should be able to review this over the weekend (bit busy till Saturday).

[dufywihi]

Great. I added the required dependencies to the script, but omitted the apt-get upgrade. That might break someone's system if they don't mean to completely upgrade certain packages.

I also left the bottom export, source and make lines alone. You can modify those as you wish, I'm not entirely positive I have the order correct. Let me know if you'd like me to make changes or add to the script, I don't mind working on this further.

#!/bin/bash
    # make sure installer script isnt run as root
    if [ "$(id -u)" = "0" ]; then
    echo -e "\nThis script must not be run as root\n" 1>&2;
    exit 1;
    # debian usage warning
    if [[ $(lsb_release -is) != 'Debian' ]]; then
        echo -e "\nThis script was designed for Debian. Press [Enter] to proceed. Otherwise, press Ctrl+c to exit installer.\n";
        read;
    fi;
    fi;
    # update apt index
    sudo apt-get update;

    # check for chomper dir, in case script is run multiple times
    if [ -d ~/chomper ]; then
    echo -e "\nChomper directory detected. Remove the directory with '$ rm -rf ~/chomper' to use this script.\n";
    exit 1;
    else
    # make sure depends are installed. only checking for zlib1g but we can probably remove the entire `if` statement and just attempt to install all the depends every time, no harm in it
    if ! [ -x "$(command -v zlib1g-dev)" ]; then
        echo -e "\nInstalling dependencies...\n";
        sudo apt-get install git build-essential curl zlib1g-dev libbz2-dev libsqlite3-dev libreadline-dev libncurses5-dev libssl-dev libgdbm-dev python-pip -y;
    fi;
    cd ~/ && git clone https://github.com/aniketpanjwani/chomper.git;
    cd chomper;
    fi;
    # shouldnt `export` and `source` come before the `make` command? you have it reversed in the README.md
    export PATH="/home/$USER/chomper/bin:$PATH" >> ~/.bashrc;
    source /home/$USER/.bashrc;
    make init

[aniketpanjwani]

This looks good. I'm thinking through a few things.

1. Not obvious to me that it matters if putting the executable on PATH comes before or after make command.
2. Given we have this shell script, what's the point of the makefile init rule? You could just more or less copy and paste the init rule into here.
3. Ideally, I'd like to introduce some sort of control flow that checks Linux distribution or OSX version and is able to do a different installation without changing the installer command. So, we could have a single installation command, which installs Chomper regardless of your non-Windows OS.
4. Ideally, we'd also have some way of uninstalling cleanly. We're not really tossing files around the system, so all you'd have to do is remove Chomper from path and make sure the virtualenv gets deleted.

I'm not close to an expert application developer, so I need to do some research and think about what the best installation architecture would be - Makefile, shell script, one or both. If you've got any thoughts, I'd greatly appreciate them.

[aniketpanjwani]

Some more semi-random thoughts:

1. It seems like the Unix standard for cross-platform installation is to have a ./config file which checks for dependencies, then use make to compile your application, and make install to put your application files where you want them. I suppose that since Python is interpreted, you would skip the make step with a Python application. Worth looking into.
2. One option might be to provide an automatic installation option only for Debian users. Everyone else has to do a bit of manual work to install, but we can put the manual work in the docs. I think I've seen this type of suggestion in a few other applications.
3. Eventually, I'd like to have just a single executable (prob using pyinstaller). If I have that, you only need to do certificate installation, enabling packet forwarding, and installation of dependencies. However, maybe it would be possible to initiate certificate installation/packet forwarding/installation of dependencies in the executable the very first time it is run. If they're not done, you could do those things.

[aniketpanjwani]

Overall, I think the development path should be

1. Create an automated shell script only for debian users (which you've already done, but where should we put it? in this repo? in a separate chomper-installation repo with shell scripts for different distros? probably just in a gist for now).
2. Create a section in the docs on how to do manual installation on other Linux distros.
3. Use pyinstaller to freeze the application, and update the docs for that.
4. Port Chomper to OSX.
5. Create a GUI.

[dufywihi]

Hmm, yeah, I'd keep it as a gist. Then in your README have something like:

Debian users can install Chomper with a single command using the installer script.

$ curl -sL https://gist.github.com/aniketpanjwani/path/to/installer.sh | bash -

For manually installation, follow these steps:

1. ...
2. ...

Or use wget as an alternative? Totally up to you.

$ wget -qO- https://gist.github.com/aniketpanjwani/path/to/installer.sh | bash -

As far as OSX and GUI development, that's a little beyond my skill as a coder :)

Also, keep in mind, I'm only trying to get this installation to as few commands as possible -- which I still haven't been able to do: https://github.com/aniketpanjwani/chomper/issues/30 After that, I'll submit my article to tecmint. I'm not sure I'll be able to contribute to chomper much after that, as I'll be working on other articles and projects.

[aniketpanjwani]

Yes - that's what I had in mind with the gist. And sorry - those comments were just clarifying my own thoughts - I wasn't expecting you to continue on with other development. You've been a great help with your work on the installation script alone.

I'm going to work on the installation script today, but if I'm reading your comment correctly, the shell script does not work just yet?

[aniketpanjwani]

Made some edits to the script. Going to test it in an AWS instance

#!/bin/bash

# make sure installer script isnt run as root
if [ "$(id -u)" = "0" ]
then
    echo -e "\nThis script must not be run as root\n" 1>&2
    exit 1

  # debian usage warning
  if [[ $(lsb_release -is) != 'Debian' ]]
  then
    echo -e "\nThis script was designed for Debian-based distributions. Press [Enter] to proceed. Otherwise, press Ctrl+c to exit installer.\n";
    read;
  fi
fi

# update apt index
sudo apt-get update;

# check for chomper dir, in case script is run multiple times
if [ -d ~/chomper ]
then
    echo -e "\nChomper directory detected. Remove the directory with '$ rm -rf ~/chomper' to use this script.\n";
    exit 1;
else
  echo -e "\nInstalling dependencies...\n";
  sudo apt-get install git build-essential curl zlib1g-dev libbz2-dev libsqlite3-dev libreadline-dev libncurses5-dev libssl-dev libgdbm-dev python-pip -y;
fi

cd ~/ && git clone https://github.com/aniketpanjwani/chomper.git;
cd chomper;

if [ ":$PATH:" != *":/home/$USER/chomper/bin"* ]
then
  echo 'export PATH=$PATH:/home/$USER/chomper/bin' >> ~/.bashrc
  echo -e "\nAdded Chomper to PATH.\n"
else
  echo -e "\nChomper is already on PATH.\n"
fi
. /home/$USER/.bashrc;
make init

[aniketpanjwani]

Close to finished - working on it here: https://gist.github.com/aniketpanjwani/bab67be0e685b65c13a6ec1cc132e321

[aniketpanjwani]

I've got it working on an AWS ubuntu instance. Just need to do the following command:

curl -sL https://gist.githubusercontent.com/aniketpanjwani/bab67be0e685b65c13a6ec1cc132e321/raw/a7c8bc9a19c926270ed48256e5818ccdbf49ebc4/chomper_install.sh | bash -

Might want to use bit.ly to shorten that URL.

[aniketpanjwani]

Found some a URL Shortener at git.io, so you can instead just use the following:

curl -sL https://git.io/vxORB | bash

Let me know if this works for you. If so, I'll update the docs with new installation instructions, and then close these two issues.

[aniketpanjwani]

@dufywihi - did you get a chance to look at this?

[dufywihi]

Sorry for the delay, been caught up in other projects.

Still not quite there...

~$ curl -sL https://git.io/vxORB | bash

Hit:1 http://security.debian.org/debian-security stretch/updates InRelease
Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease   
Hit:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Hit:4 http://cdn-fastly.deb.debian.org/debian stretch Release
Reading package lists... Done

Installing dependencies...

Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.3).
libbz2-dev is already the newest version (1.0.6-8.1).
libgdbm-dev is already the newest version (1.8.3-14).
git is already the newest version (1:2.11.0-3+deb9u2).
libncurses5-dev is already the newest version (6.0+20161126-1+deb9u2).
libssl-dev is already the newest version (1.1.0f-3+deb9u1).
python-pip is already the newest version (9.0.1-2).
libreadline-dev is already the newest version (7.0-3).
screen is already the newest version (4.5.0-6).
libsqlite3-dev is already the newest version (3.16.2-5+deb9u1).
zlib1g-dev is already the newest version (1:1.2.8.dfsg-5).
curl is already the newest version (7.52.1-5+deb9u5).
The following packages were automatically installed and are no longer required:
  linux-headers-4.9.0-4-amd64 linux-headers-4.9.0-4-common linux-image-4.9.0-3-amd64 linux-image-4.9.0-4-amd64
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  libnss3-tools
0 upgraded, 1 newly installed, 0 to remove and 83 not upgraded.
Need to get 856 kB of archives.
After this operation, 4,148 kB of additional disk space will be used.
Get:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libnss3-tools amd64 2:3.26.2-1.1+deb9u1 [856 kB]
Fetched 856 kB in 6s (140 kB/s)                                                                                                        
Selecting previously unselected package libnss3-tools.
(Reading database ... 224068 files and directories currently installed.)
Preparing to unpack .../libnss3-tools_2%3a3.26.2-1.1+deb9u1_amd64.deb ...
Unpacking libnss3-tools (2:3.26.2-1.1+deb9u1) ...
Setting up libnss3-tools (2:3.26.2-1.1+deb9u1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Cloning into 'chomper'...
remote: Counting objects: 406, done.
remote: Compressing objects: 100% (83/83), done.
remote: Total 406 (delta 85), reused 148 (delta 72), pack-reused 237
Receiving objects: 100% (406/406), 885.79 KiB | 174.00 KiB/s, done.
Resolving deltas: 100% (221/221), done.

Added Chomper to PATH.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100   148  100   148    0     0     42      0  0:00:03  0:00:03 --:--:--    42
100  2105  100  2105    0     0    287      0  0:00:07  0:00:07 --:--:--   632

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/dufy/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Requirement already up-to-date: pipenv in /usr/local/lib/python2.7/dist-packages
Requirement already up-to-date: requests[security]; python_version < "3.0" in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: pip>=9.0.1 in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: virtualenv-clone>=0.2.5 in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: setuptools>=36.2.1 in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: virtualenv in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: ordereddict; python_version < "3.0" in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: certifi in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: pathlib; python_version < "3.4" in /usr/local/lib/python2.7/dist-packages (from pipenv)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: pyOpenSSL>=0.14; extra == "security" in /usr/local/lib/python2.7/dist-packages (from requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: cryptography>=1.3.4; extra == "security" in /usr/local/lib/python2.7/dist-packages (from requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: six>=1.5.2 in /usr/local/lib/python2.7/dist-packages (from pyOpenSSL>=0.14; extra == "security"->requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: cffi>=1.7; platform_python_implementation != "PyPy" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4; extra == "security"->requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: enum34; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography>=1.3.4; extra == "security"->requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: asn1crypto>=0.21.0 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4; extra == "security"->requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: ipaddress; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4; extra == "security"->requests[security]; python_version < "3.0"->pipenv)
Requirement already up-to-date: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.3.4; extra == "security"->requests[security]; python_version < "3.0"->pipenv)
Warning: Python 3.6.4 was not found on your system…
You can specify specific versions of Python with:
  $ pipenv --python path/to/python
/home/dufy/.local/lib/python2.7/site-packages/pipenv/utils.py:1152: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/pipenv-lL_Jdu-requirements'>
  warnings.warn(warn_message, ResourceWarning)
x509: Cannot open input file /home/dufy/.mitmproxy/mitmproxy-ca.pem, No such file or directory
x509: Use -help for summary.
cp: cannot stat '/home/dufy/.mitmproxy/mitmproxy-ca.crt': No such file or directory
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
certutil:  unable to open "/usr/local/share/ca-certificates/mitmproxy-ca.crt" for reading (-5950, 2).
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Not sure what the mitmproxy error is all about:

~$ apt-cache policy mitmproxy 
mitmproxy:
  Installed: 0.18.2-6
  Candidate: 0.18.2-6
  Version table:
 *** 0.18.2-6 500
    500 http://deb.debian.org/debian stretch/main amd64 Packages
    100 /var/lib/dpkg/status

Here's what gets added to my .bashrc now:

export PATH=$PATH:/home/$USER/chomper/bin
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"

if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
fi

Running chomper in a new terminal:

~$ chomper allon 10
No virtualenv has been created for this project yet!
sudo: /bin/python: command not found

~$ chomper coding 10
No virtualenv has been created for this project yet!
sudo: /bin/python: command not found

Also maybe consider adding something like this to the installer script:

# chomper PATH detection
if ! grep -i 'chomper' ~/.bashrc 2>&1 >/dev/null; then
    echo -e "\n[+] Adding chomper to PATH.\n"
    # add export/PATH stuff here
else
    echo -e "\n[!] Chomper detected in ~/.bashrc already. Skipping this step...\n"
fi

[aniketpanjwani]

1. Agreed on adding the if statetment for chomper path detection.
2. I think the entire reason we are getting problems is due to line . ~/.bashrc on line 47 failing to actually source the .bashrc. Working on fixing it.

[aniketpanjwani]

Whoops - fat finger closed it.

[aniketpanjwani]

I created a new repo for install scripts (https://github.com/aniketpanjwani/chomper_installers). Try this:

curl -sL https://raw.githubusercontent.com/aniketpanjwani/chomper_installers/master/debian.sh | bash && source ~/.bashrc

[aniketpanjwani]

Going to close this, since I think it's resolved and haven't heard about any problems.