Standardize installation and execution

[mfripp]

I think we need to standardize how SWITCH is installed and run, because we seem to have two different versions methods in use. I'll describe below how I do it, which is my recommended approach. But I would like to hear if people have other suggestions. I would then like to rewrite the instructions so users can get up and running pretty easily.

I install SWITCH by cloning the repository, then cd'ing into the switch directory and running python setup.py develop or python setup.py develop --user. This installs the package in-place, so I can edit it and use it at the same time (and if I want to change to another installation, I just go there and run python setup.py develop again). Users who don't want to edit the package can instead run python setup.py install or python setup.py install --user. Once we put the package on pypi and conda, they can just do pip install switch or conda install -c switch-model switch . These commands work well under anaconda, and would probably work with homebrew python. With the standard system python, users might need to use sudo with the system-wide versions.

These commands do a few useful things:

• install the switch_mod package in the python system path
• install dependencies specified in setup.py (Pyomo and its dependencies for now, but the list could be extended)
• install the switch command-line script (equivalent to python -m switch_mod.main)

I then setup models in various locations in my file system (not inside the switch folder) and solve them by running switch solve or switch solve-scenarios. If I edit the local copy of the switch package or use git pull, those changes are automatically reflected the next time I use the command-line script. This works well, and makes it easy to setup other users and share models with them (via separate repositories for each model).

As I understand it from the INSTALL file, other SWITCH users are editing their PYTHONPATH to point to the switch repository and then using pip install --user -r pip_requirements.txt to install the dependencies. I don't know how other users are activating the switch command-line script, if they use it at all. This approach has a few disadvantages compared to my approach, which make it difficult to give general installation instructions for new users:

• it is hard to write general instructions for users to modify their PYTHONPATH appropriately;
• using pip requires installing pip on some systems, and may cause problems for people who use anaconda
• it's nearly impossible to write cross-platform instructions to install the switch command-line script, which is very handy, especially for new users

All of these problems are addressed automatically by setuptools, which is used by setup.py. This takes care of all the cross-platform issues other than installing the solver.

Any objections to making setup.py the standard way to install switch, via the various commands listed in the second paragraph? Or making switch solve ... the standard way to run it?

[rodrigomha]

Hi everyone,

Mi name is Rodrigo, and I'm working on SWITCH with Benjamin here at the Catholic University in Chile. In my opinion this seems to be the best choice to install and run SWITCH. When I installed SWITCH, I did it using pip and everything else, and it was really troublesome to do it.

Just to let you know, my main problem was to include SWITCH on my Windows PATH. I didn't realize that my PATH has already a length of around 1000 characters and it was not possible to include the switch-master folder on it (I think there is a max of 1024 characters). I fixed it deleting some useless lines on my PATH.

The main issue is that you don't get a specific error if including the PATH doesn't work (you will know because other things simply won't work), so maybe including a note in the README or setup file about that problem will be a good idea.

Regards, Rodrigo

[bmaluenda]

I agree the setup.py procedure is easier to grasp for new users. We've had a lot of trouble setting up new users in different platforms. It is even convenient if developers switch machines or operating systems, because settling in would be quicker.

The README and INSTALL files should be modified accordingly with instructions on this setup procedure both for users and developers. Maybe the how to collaborate.txt file could be merged in one of those to have a full set of instructions for new developers.

Additionally, the package dependencies in pip_requirements.txt could be added to the setup.py script so that people don't have to manually install them.

It would be great if the installation and usage is standardized. Great initiative.

[mfripp]

What do you all think of recommending Anaconda or miniconda as the preferred way to install Python, glpk and git? With that, the instructions can be pretty streamlined, like the attached file:

readme.txt

Also note: a similar workflow can be used to setup bigger collaborative models, where the switch package is embedded as a submodule. There's an example of this here: https://github.com/switch-hawaii/main

[bmaluenda]

Anaconda has been the easiest way for me to setup SWITCH for new users, so I'm all for it.

I would add some notes on what the setup.py script actually does, so user-devs have an easier time setting up multiple Switch packages.

Josiah was considering adding some tweaks to the setup.py script, so I would wait for his comments before posting the new instructions.

PS: I think we need to get a website up.

[josiahjohnston]

I really like the idea of streamlining installation and usage. Thank you for putting this together.

See pull request #74 for my attempt at implementation. I had issues with using setup.py directly, so I wrote a draft INSTALL to use setup.py in conjunction with pip based on the Python Packaging User Guide. See details below.

Using switch as the main entry point sounds great, but for me I couldn't get this script if I included the --user flag. This is true whether I use python setup.py develop --user or pip install --editable --user . I haven't found a workaround for that, and have only tested it on a mac so far.

Directing python newbies to Anaconda seems fine, and veteran python developers can use whatever setup they like. In either case, we should be able to assume that pip is available, since pip is included in Anaconda and Miniconda and pip is a core tool of python developers.

We definitely need setuptools and setup.py since they are needed for pip, but I'm concerned about setup.py being the main entry point because all of the python packaging advice I read recommends using pip. For non-developers, distributing on PyPi / conda should be the end-game, I think. For developers, running pip install --editable . instead of python setup.py develop should be recommended, possibly using virtual environments. pip will parse setup.py, but does a better job at managing dependencies, and allows you to uninstall the package if desired (unlike setup.py by itself). Pip also has support for "extras" like the R or psql interfaces (unlike setup.py by itself)

When I first tried to use setup.py on my computer, it totally mangled all of my python dependencies in a way that I had never seen before, and it didn't give me a clean way to undo its mess. I ended up removing homebrew and anaconda, upgrading my OS to Sierra to get an os-distribution of python that was semi-modern, and trying again from a relatively clean slate. I was able to get it working after that, but the experience makes me cautious about recommending setup.py to developers with complex setups. If everyone else can test setup.py

I like the details in the readme file you wrote. Could we use it for INSTALL.txt rather than README? One comment is to mention Cbc in the list of solvers; it's open source and a lot faster than glpk.

References I found useful:

• Python Packaging User Guide: Packaging and Distributing Projects
• install_requires vs Requirements files

[mfripp]

I agree, we should make pip install --editable . the standard installation method (or eventually conda for non-developer users). I was already leaning that way, and this seals the deal. I just posted another pull request involving some changes to setup.py, and I'll go back and revise that soon.

When using setup.py directly, I think python setup.py develop --user is useful for HPC systems, where it installs the command-line scripts somewhere in the user path, and doesn't require system-level access. It may also be useful on Unixy systems running the system-installed Python. But the '--user' flag is unnecessary with the conda python, and I think it may install the scripts in some location under ~/.local that is not in the standard search path. If we move to pip, this may not be an issue. Or it may be. I'll see if I can come up with some simple instructions that cover the normal range of cases.

I had a cbc installation in my readme.txt at one point. With conda, you just need to run conda install -c conda-forge coincbc. However, no one provides a conda package for a Windows build of coincbc, and coincbc solved my test case slower than glpk, so I removed it to keep things simple. I can add it back as an optional step, with notes.