NuPIC should be pip-installable

[chetan51]

We should have a setup.py file that calls cmake, doesn't require any environment variables to be set by the user, and installs NuPIC in a pip/python standard way.

[utensil]

:+1: Related to #805 .

[utensil]

At http://lists.numenta.org/pipermail/nupic-hackers_lists.numenta.org/2014-April/000531.html , @david-ragazzi suggested a directory structure like:

├── setup.py
├── CONTRIBUTING.md
├── Dockerfile
├── Doxyfile
├── LICENSE.txt
├── README.md
├── bin/
│   ├── run_swarm.py
│   └── run_tests.py
├── docs/
├── examples/
│   ├── audiostream/
│   ├── bindings/
│   ├── opf/
│   ├── prediction/
│   ├── sp/
│   ├── swarm/
│   └── tp/
├── tests
│   ├── __init__.py
│   ├── external/
│   ├── htmtest/
│   ├── integration/
│   ├── regression/
│   ├── shared_data/
│   ├── swarming.testlist
│   └── unit/
├── nupic/
│   ├── core/
│   │   ├── __init__.py
│   │   ├── algorithms/
│   │   ├── data/
│   │   ├── database/
│   │   ├── encoders/
│   │   ├── frameworks/
│   │   ├── image/
│   │   ├── math/
│   │   ├── research/
│   │   ├── support/
│   │   ├── swarming/
│   │   └── test/
│   └── regions/
│       ├── __init__.py
│       ├── CLAClassifierRegion.py
│       ├── CLARegion.py
│       ├── CLARegionSimple.py
│       ├── ImageSensor.py
│       ├── ImageSensorExplorers/
│       ├── ImageSensorFilters/
│       ├── KNNAnomalyClassifierRegion.py
│       ├── KNNClassifierRegion.py
│       ├── PCANode.py
│       ├── PictureSensorExplorers/
│       ├── PyRegion.py
│       ├── RecordSensor.py
│       ├── RecordSensorFilters/
│       ├── SPRegion.py
│       ├── SVMClassifierNode.py
│       ├── Spec.py
│       ├── TPRegion.py
│       ├── TestNode.py
│       ├── TestRegion.py
│       ├── UnimportableNode.py
│       └── extra/
└── extensions/
    ├── nupic_bindings/
    │   ├── build/
    │   │   ├── release/
    │   │   │   ├── bin/
    │   │   │   │   └── testpyhtm
    │   │   │   └── lib/
    │   │   │       ├── _algorithms.so
    │   │   │       ├── _engine_internal.so
    │   │   │       ├── _iorange.so
    │   │   │       ├── _math.so
    │   │   │       ├── algorithms.py
    │   │   │       ├── engine_internal.py
    │   │   │       ├── iorange.py
    │   │   │       └── math.py
    │   │   └── scripts/
    │   └── src/
    │       ├── CMakeLists.txt
    │       ├── common/
    │       │   └── engine_internal_common.i
    │       ├── cs/
    │       ├── java/
    │       └── py/
    │           ├── numpy.i/
    │           ├── reals.i/
    │           ├── types.i/
    │           ├── engine_internal/
    │           │   └── engine_internal_py.i
    │           ├── algorithms/
    │           │   ├── algorithms_impl.i
    │           │   └── algorithms_py.i
    │           ├── iorange/
    │           │   ├── WrappedVector.hpp
    │           │   └── iorange_py.i
    │           └── math/
    │               ├── PySparseTensor.cpp
    │               ├── PySparseTensor.hpp
    │               ├── SparseMatrix.i
    │               ├── SparseTensor.i
    │               └── math_py.i
    ├── nupic_core/
    │   ├── CONTRIBUTING.md
    │   ├── Doxyfile
    │   ├── README.md
    │   ├── build/
    │   │   ├── release/
    │   │   │   ├── bin/
    │   │   │   │   ├── testcpphtm
    │   │   │   │   └── testeverything
    │   │   │   ├── include/
    │   │   │   │   └── nta
    │   │   │   │       ├── algorithms/
    │   │   │   │       ├── engine/
    │   │   │   │       ├── math/
    │   │   │   │       ├── ntypes/
    │   │   │   │       ├── os/
    │   │   │   │       ├── regions/
    │   │   │   │       ├── test/
    │   │   │   │       ├── types/
    │   │   │   │       └── utils/
    │   │   │   └── lib
    │   │   │       └── libnupic_core.a
    │   │   └── scripts/
    │   ├── doc/
    │   ├── external/
    │   └── src/
    │       ├── CMakeLists.txt
    │       ├── main
    │       │   ├── algorithms/
    │       │   ├── engine/
    │       │   ├── math/
    │       │   ├── ntypes/
    │       │   ├── os/
    │       │   ├── regions/
    │       │   ├── test/
    │       │   ├── types/
    │       │   └── utils/
    │       └── test
    │           ├── htmtest/
    │           └── testeverything/
    ├── nupic_cpp_region/
    │   ├── README
    │   ├── build/
    │   │   ├── release/
    │   │   │   └── lib/
    │   │   │       └── libcpp_region.dylib
    │   │   └── scripts/
    │   └── src/
    │       ├── CMakeLists.txt
    │       ├── main/
    │       │   ├── PyRegion.cpp
    │       │   └── PyRegion.hpp
    │       └── test/
    │           ├── PyHelpersTest.cpp
    │           └── PyHelpersTest.hpp
    └── python_support/
        ├── README
        ├── build/
        │   ├── release/
        │   │   └── lib/
        │   │       └── libsupport.a
        │   └── scripts/
        └── src/
            ├── CMakeLists.txt
            ├── NumpyVector.cpp
            ├── NumpyVector.hpp
            ├── PyArray.cpp
            ├── PyArray.hpp
            ├── PyHelpers.cpp
            ├── PyHelpers.hpp
            ├── PythonStream.cpp
            ├── PythonStream.hpp
            └── WrappedVector.hpp

Reasoning:

The current Nupic structure "as is" could serve that workflow(i.e. standard python project layout to be pip-installable).. However I think some improvements could be made in order to leave it more organised... So I propose some (minor) changes: The creation of a /submodules folder when we put /nta, /bindings, and /python_support (old py/support) into it .. The reason for putting /bindings into submodules is that these swig files will be reused by other bindings projects for .Net, Java, etc... but this doesn't mean that we should create a repo only for it!

[david-ragazzi]

Yeah.. I'm thinking in take this job to me.. It's good to me understand the Python stuff of NuPIC.. :-)

It's ok, @rhyolight ?

[rhyolight]

@david-ragazzi Ok with me. Ask @oxtopus and @scottpurdy your python questions.

[chetan51]

I'm excited to see this become a reality :)

[rhyolight]

It will definitely help with python installation problems. :)

---

Matt Taylor OS Community Flag-Bearer Numenta

On Mon, Apr 14, 2014 at 9:06 PM, Chetan Surpur notifications@github.comwrote:

I'm excited to see this become a reality :)

Reply to this email directly or view it on GitHubhttps://github.com/numenta/nupic/issues/809#issuecomment-40443755 .

[oxtopus]

:+1: yay!

[jcasner]

@david-ragazzi you just made my day!

[david-ragazzi]

haha.. Ok.. but I'm advising I still am newbie in python packages (althought I also was a CMake newbie).. All help will be welcome..

Unless say that we'll have to restructure the current repo to follow PIP/distools workflow.. See the directory structure quoted by @utensil . Don't worry, are minor changes...

[david-ragazzi]

I created the PR #813 which anyone can follow the work..

By the way, I'm following this tutorial: http://getpython3.com/diveintopython3/packaging.html

[oxtopus]

I think I would actually prefer something where the user would call cmake first, and the result partly being a python package (say, a binary egg or platform-specific wheel) that is installable with pip and/or included setup.py. I think this approach would be cleanest and give the most flexibility and paves the way to binary releases, which wouldn't require the user to have to build anything themselves if they aren't going to be developing for NuPIC but instead will be just users.

[david-ragazzi]

I think I would actually prefer something where the user would call cmake first, and the result partly being a python package (say, a binary egg or platform-specific wheel) that is installable with pip and/or included setup.py. I think this approach would be cleanest and give the most flexibility and paves the way to binary releases, which wouldn't require the user to have to build anything themselves if they aren't going to be developing for NuPIC but instead will be just users.

I'm working to user pass extra flags to CMake in python setup.py install command.. setup.py will handle the command line to get these options:

http://stackoverflow.com/questions/677577/distutils-how-to-pass-a-user-defined-parameter-to-setup-py

This way the process will be the very easy, because everything will can be done using a single line using setup.py..