Package the OMF for pip distribution [30h]

[mhavard999]

1. Make pip package. Likely wheel type.
2. Test on VMs: windows and linux.
3. Upload to Pypi.

[mhavard999]

On Windows, it seems the .pth file needs to be in the site-packages directory under the local user's AppData directory, rather than the root Python27\Lib\site-packages directory,

[mhavard999]

This article is interesting: https://caremad.io/blog/setup-vs-requirement/

[dpinney]

I got the .pth file to work installed in the system site-packages folder.

In C:\Python27\Lib\site-packages, I added a file named omf.pth with a single line in it: /Users/dwp0/Dropbox/OMF/. That's the parent directory for the omf folder. I.e. feeder.py's full path is /Users/dwp0/Dropbox/OMF/omf/feeder.py.

Then from omf import feeder works as expected.

[dpinney]

Here's the goal: https://pypi.python.org/pypi?%3Aaction=submit_form

[dpinney]

The long story: http://lucumr.pocoo.org/2014/1/27/python-on-wheels/ Quick and dirty advice: http://flask.pocoo.org/docs/patterns/distribute/ Standard library docs: https://docs.python.org/2/distutils/

[cshjin]

Which license of OMF is? GPL? http://choosealicense.com/

[dpinney]

No license decision yet. Don't specify a license.

[cshjin]

Currently we can install omf from distribution zip file, which include setup.py and other related configurations. Two problem left:

1. failures happened when install numpy and matplotlib. It usually happened when using pip to install such packages.
2. DLL missing, like #186, must install MSVCP110.dll.

[cshjin]

Here is the preliminary procedure of installing omf on Linux

1. Check the version of python

   crn@debian:~$ python 
   Python 2.7.3 (default, Mar 13 2014, 11:03:55) 
   [GCC 4.7.2] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> 

2. Install pip

   crn@debian:~$ sudo python get-pip.py 
   [sudo] password for crn: 
   Downloading/unpacking pip
    Downloading pip-1.5.6-py2.py3-none-any.whl (1.0MB): 1.0MB downloaded
   Downloading/unpacking setuptools
    Downloading setuptools-4.0.1-py2.py3-none-any.whl (549kB): 549kB downloaded
   Installing collected packages: pip, setuptools
   Successfully installed pip setuptools
   Cleaning up...

3. Download ZIP file and unzip it

   crn@debian:~$ wget <some path or server>/omf-0.1.zip
   crn@debian:~$ unzip omf-0.1.zip 

4. Install from setup.py.

   crn@debian:~$ sudo python setup.py install
   ...
   Using /usr/lib/python2.7
   Finished processing dependencies for omf==0.1

   If there is any errors, please refer to FAQ.

5. Test omf.

   crn@debian:~/omf-0.1$ python
   Python 2.7.3 (default, Mar 13 2014, 11:03:55) 
   [GCC 4.7.2] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import omf
   >>> from omf import models

   If there is no error, which means you have installed omf successfully, check for step 4.

6. Run omf.

   $ cd omg-0.1/omf
   $ python web.py

FAQ: NOTE: the following problems happened in a Debian Linux system. The commands may differ from different distribution os Linux systems.

1. Install matplotlib

   ============================================================================
                          * The following required packages can not be built:
                          * freetype

   It is a dependency of matplotlib, and pip won't install a system-level dependencies. You need to install:

   crn@debian:~/omf-0.1$ sudo apt-get install libfreetype6-dev 

2. GCC compile:

   gcc: error trying to exec 'cc1plus': execvp: No such file or directory
   error: Setup script exited with error: command 'gcc' failed with exit status 1
   Error in atexit._run_exitfuncs:

   It is because system using gcc to compile matplotlib, instead of using g++. To fix it:

   crn@debian:~/omf-0.1$ sudo apt-get install g++

3. Python.h missing:

   ./CXX/WrapPython.h:58:20: fatal error: Python.h: No such file or directory
   compilation terminated.
   error: Setup script exited with error: command 'gcc' failed with exit status 1
   Error in atexit._run_exitfuncs:

   Python.h is nothing but a header file. It is used by gcc to build applications. You need to install a package called python-dev. This package includes header files, a static library and development tools for building Python modules, extending the Python interpreter or embedding Python in applications. To fix it:

   crn@debian:~/omf-0.1$ sudo apt-get install python-dev

4. Missing png libs:

   /usr/include/features.h:165:0: note: this is the location of the previous definition
   src/_png.cpp:10:20: fatal error: png.h: No such file or directory
   compilation terminated.
   error: Setup script exited with error: command 'gcc' failed with exit status 1
   Error in atexit._run_exitfuncs:

5. Nonetype error:

   File "/usr/lib/python2.7/distutils/command/config.py", line 248, in try_link
      self._check_compiler()
    File "/tmp/easy_install-w3AiAT/numpy-1.8.1/numpy/distutils/command/config.py", line 46, in _check_compiler
    File "/usr/lib/python2.7/distutils/command/config.py", line 103, in _check_compiler
      customize_compiler(self.compiler)
    File "/tmp/easy_install-CbBzw1/matplotlib-1.3.1/setupext.py", line 194, in my_customize_compiler
   TypeError: 'NoneType' object is not callable
   Error in atexit._run_exitfuncs:
   Traceback (most recent call last):
    File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
      func(*targs, **kargs)
    File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
   TypeError: 'NoneType' object is not callable
   Error in sys.exitfunc:

   It is because the when checking the system error, you need to change system encoding from ascii to utf-8. To fix it in python command:

   crn@debian:~/omf-0.1$ python 
   Python 2.7.3 (default, Mar 13 2014, 11:03:55) 
   [GCC 4.7.2] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import sys
   >>> sys.getdefaultencoding()
   'ascii'
   >>> reload(sys)
   <module 'sys' (built-in)>
   >>> sys.setdefaultencoding("UTF-8")
   >>> sys.getdefaultencoding()
   'UTF-8'

6. numpy and matplotlib error:

   Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

   It is highly recommended to install numpy and matplotlib seperately. Since the compile processing in omf installation is a bit intricate.

   $ sudo pip install numpy, matplotlib

[cshjin]

Do we need to modify omf project layout into the following:

|
├───omf
│   ├───data
│   ├───...
├───setup.py
├───requirements.txt
└───...

The purpose of changing into this is for the package list in setup.py file.

setup(
    ...
    packages = find_packages(),
   ...
)

If it is in the current layout, then the list will be: ['models', 'solvers', 'solvers.gridlabd', 'solvers.nrelsam'] While if move setup.py into upper leverl, the package list will be: ['omf', 'omf.models', 'omf.solvers', 'omf.solvers.gridlabd', 'omf.solvers.nrelsam']

Pros: it looks like a much more standard python project layout, and easy to maintain the packages. Cons: some path issues may occur

Of course, if we can find a way to fix the package list, that would be much helpful.

[cshjin]

Here is a compromised way to install and use omf. Instead of installing python packages into site-packages directory, we install a link to _sitepackages, which is to install in a development mode. We can do like the following:

$ unzip omf-0.1.zip
$ cd omf-0.1
$ python setup.py develop
$ cd omf
$ python web.py # run omf

Or use pip to install in a editable mode:

$ unzip omf-0.1.zip
$ pip install -e omf-0.1
$ cd omf-0.1/omf
$ python web.py

It will fix the path issues, since it will only create a link into site-packages(for Windows) or dist-packages(for Linux).

Check installation:

crn@debian:~$ pip show omf
---
Name: omf
Version: 0.1
Location: /home/crn/omf-0.1
Requires: Flask, Flask-Login, Jinja2, MarkupSafe, Werkzeug, argparse, backports.ssl-match-hostname, boto, networkx, nose, passlib, pyparsing, python-dateutil, six, tornado, wsgiref, numpy, matplotlib

This optional solution is both OK for Windows and Linux

[cshjin]

Check here for instructions of how to prepare distribution file and install it.

[dpinney]

Nice docs. Could you move this to the wiki? There’s a way to check out the wiki via git – let me know if this would be helpful and I can show you how to do it.

[dpinney]

Let's remove the requirement to upload to PyPI. Since this install has been tested, and the instructions are written and on the wiki, everything else is done, right?

[cshjin]

Yes, if PyPI distribution is delayed to the future, we could close it now.

[dpinney]

Awesome. Great work on this issue.