Python refactoring: ideas

[farhi]

iData and iFunc classes can inherit from multiple sub-classes, e.g.:

Master class:

• base methods and properties: Data, Aliases, Axes, UserData, meta-data, ...
• check, set, get, copy, ...

Sub classes (see https://github.com/McStasMcXtrace/iFit/blob/master/Docs/Methods.html):

• I/O: populates the object by reading files. Methods: read/write
• View: print-out, plot, subplot, edit, ...
• Math Unary methods
• Math Binary methods
• Math statistics
• Math integration, derivatives, projections
• Math conv, corr, FFT
• Math combine, cat, dog, slice, ...
• Math interpolation

Then we assemble

• iData(Master, I/O, Math, ...)

which builds all bits and allow each sub-class to be rather small.

[farhi]

A good alternative is to use imatlab which brings a Matlab into Jupiter:

• https://github.com/imatlab/imatlab

which can probably be made into a Debian package.

[farhi]

Install for Ubuntu 18.04, python 3.6 (from system), matlab R2017a:

• cd /opt/MATLAB/R2017a/extern/engines/python/
• add '3.6' into setup.py / supported versions
• optional: cp /opt/MATLAB/R2017a/extern/engines/python/dist/matlab/engine/glnxa64/matlabengineforpython3_5.so /opt/MATLAB/R2017a/extern/engines/python/dist/matlab/engine/glnxa64/matlabengineforpython3_6.so
• optional: sudo ghex /opt/MATLAB/R2017a/extern/engines/python/dist/matlab/engine/glnxa64/matlabengineforpython3_6.so
• optional: change all '3_5' occurences to '3_6', and '3.5' to '3.6'. Save.
• sudo python3.6 setup.py install
• modify /usr/local/lib/python3.6/dist-packages/matlab/engine/init.py to add '3_6' in supported versions.
• modify /usr/local/lib/python3.6/dist-packages/matlab/engine/init.py to explicitely explicitely import_module("matlabengineforpython3_5") [2 locations]
• sudo apt install python3-plotly python3-widgetsnbextension python3-nbconvert python3-ipykernel python3-testpath
• sudo pip3 install imatlab
• sudo python3 -mimatlab install

Installation is done in:

• matlab engine: /opt/MATLAB/R2017a/extern/engines/python
• imatlab: /usr/local/lib/python3.6/dist-packages/imatlab

[farhi]

Notes above from:

• https://github.com/imatlab/imatlab
• https://gist.github.com/dangom/303879ca649f7a3fe113912d7ffa62d4

Works OK in Jupyter.

[farhi]

Matlab Engine R2017a Installation output:

# cd /opt/MATLAB/R2017a/extern/engines/python
# sudo python3.6 setup.py install
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/matlab
copying dist/matlab/__init__.py -> build/lib/matlab
copying dist/matlab/mlexceptions.py -> build/lib/matlab
copying dist/matlab/mlarray.py -> build/lib/matlab
creating build/lib/matlab/engine
copying dist/matlab/engine/engineerror.py -> build/lib/matlab/engine
copying dist/matlab/engine/__init__.py -> build/lib/matlab/engine
copying dist/matlab/engine/matlabengine.py -> build/lib/matlab/engine
copying dist/matlab/engine/basefuture.py -> build/lib/matlab/engine
copying dist/matlab/engine/matlabfuture.py -> build/lib/matlab/engine
copying dist/matlab/engine/futureresult.py -> build/lib/matlab/engine
copying dist/matlab/engine/enginesession.py -> build/lib/matlab/engine
copying dist/matlab/engine/fevalfuture.py -> build/lib/matlab/engine
creating build/lib/matlab/_internal
copying dist/matlab/_internal/__init__.py -> build/lib/matlab/_internal
copying dist/matlab/_internal/mlarray_utils.py -> build/lib/matlab/_internal
copying dist/matlab/_internal/mlarray_sequence.py -> build/lib/matlab/_internal
running install_lib
creating /usr/local/lib/python3.6/dist-packages/matlab
creating /usr/local/lib/python3.6/dist-packages/matlab/_internal
copying build/lib/matlab/_internal/__init__.py -> /usr/local/lib/python3.6/dist-packages/matlab/_internal
copying build/lib/matlab/_internal/mlarray_utils.py -> /usr/local/lib/python3.6/dist-packages/matlab/_internal
copying build/lib/matlab/_internal/mlarray_sequence.py -> /usr/local/lib/python3.6/dist-packages/matlab/_internal
copying build/lib/matlab/__init__.py -> /usr/local/lib/python3.6/dist-packages/matlab
copying build/lib/matlab/mlexceptions.py -> /usr/local/lib/python3.6/dist-packages/matlab
copying build/lib/matlab/mlarray.py -> /usr/local/lib/python3.6/dist-packages/matlab
creating /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/engineerror.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/__init__.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/matlabengine.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/basefuture.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/_arch.txt -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/matlabfuture.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/futureresult.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/enginesession.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
copying build/lib/matlab/engine/fevalfuture.py -> /usr/local/lib/python3.6/dist-packages/matlab/engine
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/_internal/__init__.py to __init__.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/_internal/mlarray_utils.py to mlarray_utils.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/_internal/mlarray_sequence.py to mlarray_sequence.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/__init__.py to __init__.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/mlexceptions.py to mlexceptions.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/mlarray.py to mlarray.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/engineerror.py to engineerror.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/__init__.py to __init__.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/matlabengine.py to matlabengine.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/basefuture.py to basefuture.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/matlabfuture.py to matlabfuture.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/futureresult.py to futureresult.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/enginesession.py to enginesession.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/matlab/engine/fevalfuture.py to fevalfuture.cpython-36.pyc
running install_egg_info
Writing /usr/local/lib/python3.6/dist-packages/matlabengineforpython-R2017a.egg-info

Test with:

# ipython3
import matlab.engine
eng = matlab.engine.start_matlab()

[farhi]

imatlab installation output [https://github.com/imatlab/imatlab] :

# sudo pip3 install imatlab
Collecting imatlab
  Downloading https://files.pythonhosted.org/packages/92/4d/ddd1e0df607266826ab2c3fe6641e3daaa7ca917c906595e362413894fcb/imatlab-0.3-py3-none-any.whl
Requirement already satisfied: widgetsnbextension>=1.0 in /usr/lib/python3/dist-packages (from imatlab)
Requirement already satisfied: nbconvert>=4.2 in /usr/lib/python3/dist-packages (from imatlab)
Requirement already satisfied: plotly>=1.13.0 in /usr/lib/python3/dist-packages (from imatlab)
Requirement already satisfied: ipykernel>=4.1 in /usr/lib/python3/dist-packages (from imatlab)
Requirement already satisfied: matlabengineforpython>=R2016b in /usr/local/lib/python3.6/dist-packages (from imatlab)
Requirement already satisfied: testpath in /usr/lib/python3/dist-packages (from nbconvert>=4.2->imatlab)
Installing collected packages: imatlab
Successfully installed imatlab-0.3
# sudo python3 -mimatlab install
Installed kernelspec imatlab in /usr/local/share/jupyter/kernels/imatlab

Test by starting Jupyter and a Matlab kernel (pop-up 'New' menu, top-right). You can then type e.g.:

• imatlab_export_fig('print-png')
• plot(1:10)

will display the static graphics in the Jupyter notebook session.

[farhi]

Configure Jupyter and apache for distant connexion:

• https://lib.smgr.io/remote-jupyter-notebook-server/

This includes a user/passwd login.

[farhi]

Installation of PlotLy (only once to set the credentials):

• download and extract plotly from e.g. https://plot.ly/matlab/getting-started/
• go there
• in a browser, login on plotly (https://plot.ly e.g. with GitHub or Facebook) and get your username and api_key from the Settings page.
• matlab> plotlysetup('UserName', 'api_key')

Setting offline plotting (at every Matlab startup):

• if behind a firewall, set matlab proxy (I use 'webbrowser' and its Navigation/proxy menu item -> dialog)
• matlab> getplotlyoffline('http://cdn.plot.ly/plotly-latest.min.js'); NOTE: http and not https if an SSL error occurs.
• this last command should be used at Matab startup.

Generating plotly figures:

• matlab> plot(1:10); fig2plotly(gcf, 'offline', true);
• To preserve the original MATLAB color and styling, use: fig2plotly(gcf,'strip',false,'offline',true)
• this last command plot within Matlab, and also open a local PlotLy interactive view in a browser
• simple plot are OK, volume plots are not rendered.

[farhi]

Accessing Notebook from 'outside' (not localhost):

• https://stackoverflow.com/questions/42848130/why-i-cant-access-remote-jupyter-notebook-server

indicates that the configuration file has to allow distant connection in file:

• ~/.jupyter/jupyter_notebook_config.py

then changes lines:

• c.NotebookApp.allow_origin = '*' #allow all origins
• c.NotebookApp.ip = '0.0.0.0' # listen on all IPs

Adding authentication (basic)

To add a password, use:

• jupyter notebook password

will modify the above configuration to store the password as a hash (encoded).

[farhi]

An other solution is to write a Jupyter Kernel around iFit runtime executable:

• https://jupyter-client.readthedocs.io/en/stable/wrapperkernels.html

seems rather simple to set-up.

[farhi]

The above example does not start the kernel. An example that works properly:

• https://github.com/jupyter/echo_kernel

Installation:

# sudo python3 setup.py install
/usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/echo_kernel
copying echo_kernel/__init__.py -> build/lib/echo_kernel
copying echo_kernel/install.py -> build/lib/echo_kernel
copying echo_kernel/kernel.py -> build/lib/echo_kernel
copying echo_kernel/__main__.py -> build/lib/echo_kernel
running install_lib
creating /usr/local/lib/python3.6/dist-packages/echo_kernel
copying build/lib/echo_kernel/__init__.py -> /usr/local/lib/python3.6/dist-packages/echo_kernel
copying build/lib/echo_kernel/install.py -> /usr/local/lib/python3.6/dist-packages/echo_kernel
copying build/lib/echo_kernel/kernel.py -> /usr/local/lib/python3.6/dist-packages/echo_kernel
copying build/lib/echo_kernel/__main__.py -> /usr/local/lib/python3.6/dist-packages/echo_kernel
byte-compiling /usr/local/lib/python3.6/dist-packages/echo_kernel/__init__.py to __init__.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/echo_kernel/install.py to install.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/echo_kernel/kernel.py to kernel.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/echo_kernel/__main__.py to __main__.cpython-36.pyc
running install_egg_info
Writing /usr/local/lib/python3.6/dist-packages/echo_kernel-1.1.egg-info
# sudo python3 -m echo_kernel.install
Installing Jupyter kernel spec

Locations:

• echo kernel in /usr/local/share/jupyter/kernels/echo
• echo_kernel package in /usr/local/lib/python3.6/dist-packages/echo_kernel with a dummy __init.py__ and __main.py__. The actual kernel is in kernel.py.

Actually, there is also a simple plotter kernel:

• https://ipython-books.github.io/16-creating-a-simple-kernel-for-jupyter/

which mentions the format of the messaging protocol:

• https://jupyter-client.readthedocs.io/en/latest/messaging.html#execution-results
• https://buildmedia.readthedocs.org/media/pdf/jupyter-client/latest/jupyter-client.pdf