Blendyn with Blender's own Python and the required packages

[louisgag]

This issue can count towards the entry exam for the Blendyn related projects for MBDyn-GSoC2017 This issue may be quite challenging and not much support can be given by the mentors

In its current form, the Blendyn plugin needs a series of Python packages such as numpy, netCDF4, pygal, cairosvg, tinycss, lxml, cssselect, etc. to run with full potential. These packages are not part of the official Blender binaries that are released on the official Blender website. Furthermore, the pip interface required to install Python packages is not available in the Python that comes with precompiled versions of Blender.

This means that only the system's installed Blender, which uses the system's installed Python, which in turn has the pip package manager can be used adequately with Blendyn.

A solution would be useful for all those who want to use the latest Blender version with Blendyn.

As stated above, it may be a challenging and long task as we have already attempted solutions such as installing precompiled Blenders from different sources, compiling Blender, using the system's Python with a precompiled Blender, copying/linking the interested packages from the system's Python into Blender's Python and none of those solutions worked properly. Good luck.

[janga1997]

@louisgag @zanoni-mbdyn I think this issue can be solved in a simple way. First, having conda , which helps to manage several python versions simultaneously is crucial. Now , my default conda python version is 3.5.2, which happens to be the packaged python version of the blender zip file I downloaded (Blender 2.78c), and the system default version I installed which is Blender 2.69. (Uses Python 3.4.3) I created a conda virtualenv for the version Python 3.4.3, which is easy to do.

1. Create /home/user/blender/scripts/2.69/ and /home/janga/blender/scripts/2.78c.
2. Create three directories modules/, addons/, and startup/ for in each of the above version directories.
3. In modules/, use symlink to link whatever libraries you need from the anaconda distribution, by ls -s /home/janga/anaconda3/lib/python3.5/site-packages/lxml/ lxml ls -s /home/janga/anaconda3/lib/python3.5/site-packages/scipy/ scipy
4. For the system version 2.69, since I installed a venv of 3.4.3 ln -s /home/janga/anaconda3/envs/py34/lib/python3.4/site-packages/lxml/ lxml ln -s /home/janga/anaconda3/envs/py34/lib/python3.4/site-packages/scipy/ scipy
5. In each Blender version, go over to File tab in User Preferences, and in the row of Scripts, add /home/janga/scripts/blender/{blender-version}/ The modules should be imported. You can check it typing import {module} into the Python console of Blender. And last, for users, just install Blender through zip, but for devs, add a symlink to each addons/ in /home/janga/blender/scripts/{blender-version}, and check mark to import Blendyn.

It works pretty smoothly, and installing new packages through conda is a breeze. Since the default and venvs of conda come with pip too, we can also use pip to install modules in conda, and then add the symlinks.

This solution works for both the extracted and the installed versions.

[janga1997]

I think the key thing here is to that python versions should match exactly. And for that, we need something like conda.

[zanoni-mbdyn]

Thank @janga1997, this can be good solution. Do you think it can be automated?

We could add an operator in the TOOLS region that reads something like Install optional Python Packages and is able to recognize (if that is possible...) that the version of Blender the user is running is a bundled one or coming from a distribution package.

At the very least, I think this procedure should go into the wiki, in the Installation page, but the used should be also informed that additional packages were not found at the addon first start, in the Blender UI.

Cheers Andrea

[janga1997]

I don't know how we could automate this for users. The user is probably going to just download the zip file and import it through blender.

[zanoni-mbdyn]

I was referring to the process of installing the optional packages and creating the symlinks that you mentioned in your proposed solution.

[janga1997]

Well, stuff we could automate is the creation of directories, and adding symlinks. But for installing packages with the correct version of python, otherwise we have to install a new python environment first, user confirmation would be required at every step anyway.

[janga1997]

And we would have to do this with a separate script, bash/python, which the user would have to run manually. But the assumption is that that the user would just download the zip file, right?

[zanoni-mbdyn]

But for installing packages with the correct version of python, otherwise we have to install a new python environment first, user confirmation would be required at every step anyway.

Please explain a little bit more what you mean with this comment. As for user confirmation, we might ask him to start Blender from a terminal (if he hasn't done so already: we should check if it possible to recognise that) and there is where all the confirmations will be asked for.

And we would have to do this with a separate script, bash/python, which the user would have to run manually.

Why is that? Can't we just make a subprocess.call to conda or pip?

But the assumption is that that the user would just download the zip file, right?

Not necessarily. We might move towards something like:

• the user downloads Blendyn, without having installed any of the additional packages;
• he/she starts Blender from a terminal;
• he/she clicks on the Install additional Python packages button and the installation of the required packages starts.

In the last step, if it is detected that Blender is running with the system's python, the calls to pip and to the system's package manager are made. If it is too much burdensome to do so automatically, we could just inform the user that he/she needs to install a ceirtain library (I'm thinking for example to the CFFI library) using the system's package manager. If the bundled Python is found, then all the symlinks and the calls to conda (requiring the user to install it if is not found) are made.

[janga1997]

If the blender version of the user requires a python version of 3.5.2, and the user doesn't have that exactly as either the system python, or one of environments in conda, we would have to install a new python environment first, and then install the required packages.

[janga1997]

This issue now looks like a set of features to be added to blendyn, rather than just an installation method. I'm thinking i should add this to my proposal to the list of features, and keep it that way if i am not able to completely solve it by the application period.

[zanoni-mbdyn]

If the blender version [...]

Got it. We could start by warning the user, and maybe move towards doing a complete installation upon his/her request afterwards.

This issue now looks like a set of features to be added to blendyn, rather than just an installation method.

Well, kinda :)

I'm thinking i should add this to my proposal to the list of features, and keep it that way if i am not able to completely solve it by the application period.

That is perfectly okay.

Cheers Andrea

[louisgag]

@janga1997 https://github.com/zanoni-mbdyn/blendyn/issues/8#issuecomment-287482057 So the first step for a user who has not yet any Anaconda would be to install Conda on the system's Python? pip install conda

[janga1997]

@louisgag Yes. Well, any other software which lets you create virtual environments would be fine, but conda is the easiest. But I prefer to install using the installer bash file for linux. It's around 400 MB, but you never have to worry about packaging and environments again.

[louisgag]

Install it on any of the system's Python? For example, I have 2.7.6 and 3.4.3 while my Blenders have 3.4.3 (system default) and 3.5.2

[louisgag]

But I prefer to install using the installer bash file for linux. It's around 400 MB, but you never have to worry about packaging and environments again.

Where do I find it ?

[janga1997]

https://www.continuum.io/downloads Install either the Python 2.7 or Python 3.6 version. It doesn't really matter. It is platform agnostic, so that's helpful too.

[janga1997]

And to create virtual environments conda create -n {name of env} python=3.5.2 anaconda (name it something easy to remember like py35) conda create -n py34 python=3.4.3 anaconda

and to step into the virtual environment

source activate {env_name}

source activate py35 or source activate py34

And to step out of the virtual environment,

source deactivate

Any packages you install while in a virtual environment will be saved only in that environment. This will not touch your system Python installation or system installed Python packages.

[janga1997]

Since both you guys are into research, anaconda also installs by default a lot of tools like Jupyter notebooks, Jupyter qtconsole, packages like numpy, pandas, matplotlib, scipy, astropy, etc by default.

[louisgag]

@janga1997

I followed your suggested procedure but the packages still don't get recognized by Blender 2.78c I installed anaconda and two versions of python inside it. I also noticed that it is important to be careful whether to run pip or pip3 depending on which anaconda/venv is running as the other may run on the system Python. I tried both linking the packages directly into the Blender's python and linking them into the "modules addons startup" folder...

In the end I resorted to the get-pip.py http://blender.stackexchange.com/questions/56011/how-to-use-pip-with-blenders-bundled-python/56013#56013 solution which works for netCDF4 but still brings warnings, doesn't allow installing all required packages, and doesn't allow detecting pygal..

So for now I'm going back to 2.69, the system's default... I may look at this issue again during the summer.

Regards

[janga1997]

@louisgag You can just install packages with conda instead of pip. And if you follow the rest of the steps I laid out, it does work. I realize that there may be issue with using pip, I have to look into it.

Just use conda. All the packages we require can be installed with conda.

[janga1997]

conda install -c activisiongamescience pygal=2.1.1 conda install -c anaconda netcdf4=1.2.4

[louisgag]

Thanks Janga, but for now I'll stick to 2.69 which works just fine.

[janga1997]

While struggling with netCDF4 yesterday, I think I figured out an even simpler solution to our packaging woes. Regardless of whether if it is a system installation or extracted zip file, Blender will have a directory site-packages. Since I use a extracted zip file, ~/Downloads/blender-2.78/2.78/python/lib/python3.5/site-packages This is where Blender pulls its python packages from. I just deleted this directory, and added a symlink of the site-packages of my conda installation. ~/anaconda3/lib/python3.5/site-packages All the conda packages are now available in Blendyn.

This removes the need for creating a /scripts directory, and adding symlinks for each package we need. Every time we do a conda install, it will automatically be loaded to Blender.

[janga1997]

I still feel like describing a set of general instructions is better than a dedicated script for it, because there are too many factors to consider,

• Operating System (Windows, Ubuntu , Mac)
• Bash shell (for running our install script)
• The method of symlink, which I am sure has a different way of doing it in Windows.

But if we just provide instructions OS-wise and Blender file-wise, the user has greater flexibility. I still feel like most users are going to just download the zip-file of Blendyn, and import it through Blender settings. Not everyone could be bothered with doing a git clone and running a script.

[louisgag]

On 09-May-2017 11:15 AM, VSN Reddy Janga wrote:

The method of symlink, which I am sure has a different way of doing it in Windows. This has to be tested. I remember having had difficulties with the anaconda method (Blender would complain about missing packages even if they were installed in anaconda) and in the end having switched back to the system's default Blender.

[janga1997]

@louisgag Are you on Windows or Ubuntu?

If I am not wrong

Blender would complain about missing packages even if they were installed in anaconda

You are referring to trying to install netCDF4?

[louisgag]

Ubuntu

[janga1997]

@louisgag Well, sometimes some packages (like netCDF), have other packages as dependencies, which require to be in the same directory, and adding a symlink for all of them is lengthy. That's why I think a symlink for the whole site-packages directory in conda should resolve any issues.

[louisgag]

On 09-May-2017 11:35 AM, VSN Reddy Janga wrote:

That's why I think a symlink for the whole |site-packages| directory in conda should resolve any issues.

Ok, I'll give it a try and let you know.

[zanoni-mbdyn]

@janga1997

While struggling with netCDF4 yesterday, I think I figured out an even simpler solution to our packaging > woes. ...

I think that we must be very careful with this solution. When you download Blender, you are downloading a pre-packed version that uses its own Python distribution. That's why is has its own site-packages folder.

When you install Blender from the official repositories, the version of Blender in the package is built to use the system's Python.

Removing the site-packages folder and creating a symlink to the OS one is, in my opinion, a very risky business. You are effectively trying to make the Python distribution in Blender use packages that are installed for the OS Python distribution. It certainly can go well, like in your case, but I can imagine a plethora of cases in which it will go horribly wrong!

[janga1997]

@zanoni-mbdyn I should have been more clear. The conflict of Python versions between Blender and System OS is where conda comes in.

The Blender version I use has Python 3.5.2, but my system OS has Python 2.7.6 So I use conda to create a virtual env of Python 3.5.2, install the necessary packages, and then create the symlink. The need to create an appropriate virtual env for the python version of Blender is the reason I suggested conda in the beginning.

[zanoni-mbdyn]

OK, sorry I got a bit lost on this issue. So we're left with the user needing to install a virtual environment for the correct Python version, with conda, and then we could automatically install the additional packages and create the simlinks for him?

Are we sure that this is a better solution that just ask the user to install the Python packages himself, and maybe add a section in the wiki to help him create the symlinks (or whatever the equivalent will be on Windows) by himself? Maybe it is better to just update the documentation.

It is an honest question, I don't want to throw all the work done here away. I'm just trying to evaluate if it is worth going on in the direction we seem to be headed for.

[janga1997]

Are we sure that this is a better solution that just ask the user to install the Python packages himself, and maybe add a section in the wiki to help him create the symlinks (or whatever the equivalent will be on Windows) by himself? Maybe it is better to just update the documentation.

That's what I think too. I think the user could install the packages himself with some instructions in our wiki, but a virtual environment is still needed in most cases, as it would be rare that the user has the same Python version as that of Blender. Our wiki could be something like this:

OS Python same as Blender Python Steps for Linux and Windows

OS Python different from Blender Python Steps for Linux and Windows

[zanoni-mbdyn]

@janga1997 The first part, aside from the Windows part, is already in the wiki, see the link that I have posted in the previous comment. The second one is the one that we need to add.

Would you like to write down the steps? I could then test them in different OS (I can test in Fedora 24 and 25, Ubuntu 14.04 and 16.04). Also, have you got the opportunity to try the installation of Blendyn in Windows 10? That's the only version of Windows that I have (a minimal, to be honest) interest in supporting, since the Ubuntu bash can be installed quite easily in that OS.

[louisgag]

Andrea, I can take care of testing on Ubuntu 14.04 and 16.04 once the wiki page is updated. -Louis

On 10-May-2017 01:40 PM, Andrea Zanoni wrote:

@janga1997 https://github.com/janga1997 The first part, aside from the Windows part, is already in the wiki, see the link that I have posted in the previous comment. The second one is the one that we need to add.

Would you like to write down the steps? I could then test them in different OS (I can test in Fedora 24 and 25, Ubuntu 14.04 and 16.04). Also, have you got the opportunity to try the installation of Blendyn in Windows 10? That's the only version of Windows that I have (a minimal, to be honest) interest in supporting, since the Ubuntu bash can be installed quite easily in that OS.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zanoni-mbdyn/blendyn/issues/8#issuecomment-300456334, or mute the thread https://github.com/notifications/unsubscribe-auth/ALuOfYjyrRvtFPYKFl1jEM1rp32_Q-jQks5r4aIsgaJpZM4Md-pf.

-- Louis Gagnon, Ph.D. Postdoctoral fellow Dipartimento di Scienze e Tecnologie Aerospaziali Politecnico di Milano +39 02 2399 8642 http://louisgagnon.com/research/

[janga1997]

@louisgag @zanoni-mbdyn Sure. I will update the wiki by today, (both Windows and Linux). I am guessing that the process of creating a symlink will be different in Windows, but I will figure it out.

[zanoni-mbdyn]

OK, thank you Louis.

@janga1997 please write down the steps here, I'll update the wiki myself aftwerwards.

[janga1997]

Linux

1. Install conda https://conda.io/miniconda.html . Run the bash file corresponding to your OS.

2. Open up Blender, switch to Scripting Mode , and note down the Python version displayed in the Python shell.

3. If your System Python version is different from the Blender's Python, which it most likely is, follow the below steps. Otherwise skip to step 4. --- conda create -n py35 python=3.5.2 if the Blender Python version is 3.5.2. This will create a
   virtual environment with Python version 3.5.2 --- Switch to the virtual environment you created by source activate py35. To switch back, source deactivate py35

4. Install the necessary packages (switch to virtual environment before if you created one) , conda install numpy conda install netcdf4 conda install lxml conda install pygal --- Sometimes, some packages may not be available in the official conda channels. That can be easily resolved by googling conda install {packagename}, and installing through the many unofficial channels that pop up.

5. Blender used through ZIP File --- Change to the directory of the extracted zip file. (~/Downloads/blender-2.78 for example) --- Change to the directory in which site-packages is located ~/Downloads/blender-2.78/2.78/python/lib/python3.5 (2.78 is the Blender version) This is the directory in which site-packages directory is located. --- site-packages is the directory in which all the packages to be loaded in Blendyn are located. Delete this directory site-packages

   System installation of Blender --- The site-packages is located at /usr/lib/blender/? @louisgag ( I am not sure about the complete path )

6. Create a symlink of site-packages of conda into the Blender. Make sure you are still in the directory ~/Downloads/blender-2.78/2.78/python/lib/python3.5 , the directory in which you deleted the site-packages directory. --- For virtual environment ln -s ~/miniconda3/envs/py35/lib/python3.5/site-packages

   --- For default conda environment ln -s ~/miniconda3/lib/python3.5/site-packages/

7. Now, every package you install through conda install , will be directly available for import in Blender.

[louisgag]

@janga1997 just a quick comment before I try those new instructions:

If your System Python version is different from the Blender's Python, which it most likely is, follow the below steps. Otherwise skip to step 4.

and

System installation of Blender --- The site-packages is located at /usr/lib/blender/? @louisgag ( I am not sure about the complete path )

if the system's Python the same version as Blender's, wouldn't making a symlink to the system's python site-package and installing the necessary packages through pip (as showed on the wiki) be a simpler/faster approach? Are there any issues arising from not using anaconda in this case?

[janga1997]

@louisgag No issues. Although it is named as dist-packages rather than site-packages, there shouldn't be any issues. I suggested conda for the mere fact that most people are not likely to have the same Python version as Blender.

Edit: Actually dist-packages seems to be a Debian (and Ubuntu) specific convention. Rest of the Linux distributions seem to have site-packages.

[louisgag]

I still have the same issue as the one I had before. Ubuntu 14.04. blendyn: could not find netCDF4 module. NetCDF import will be disabled.

Which means I can't test plotting neither, since it requires a NetCDF import.

I first tried with my current anaconda version, then thought it may be linked to my conda installation (used python2) so I follow your instructions step by step, see my comments:

Install conda https://conda.io/miniconda.html . Run the bash file corresponding to your OS.

Got the linux64-python 3.6 version

Open up Blender, switch to Scripting Mode , and note down the Python version displayed in the Python shell.

If your System Python version is different from the Blender's Python, which it most likely is, follow the below steps. Otherwise skip to step 4. --- conda create -n py35 python=3.5.2 if the Blender Python version is 3.5.2. This will create a virtual environment with Python version 3.5.2 --- Switch to the virtual environment you created by source activate py35. To switch back, source deactivate py35

well this (deactivating) should go after having installed the conda packages

Install the necessary packages (switch to virtual environment before if you created one) , conda install numpy conda install netcdf4 conda install lxml conda install pygal --- Sometimes, some packages may not be available in the official conda channels. That can be easily resolved by googling conda install {packagename}, and installing through the many unofficial channels that pop up.

indeed, pygal will need a little "help": conda install -c ActivisionGameScience pygal and also, for ploting, conda install -c ActivisionGameScience cairosvg conda install -c conda-forge tinycss conda install cssselect

Blender used through ZIP File --- Change to the directory of the extracted zip file. (~/Downloads/blender-2.78 for example) --- Change to the directory in which site-packages is located ~/Downloads/blender-2.78/2.78/python/lib/python3.5 (2.78 is the Blender version) This is the directory in which site-packages directory is located. --- site-packages is the directory in which all the packages to be loaded in Blendyn are located. Delete this directory site-packages

System installation of Blender
--- The site-packages is located at /usr/lib/blender/? @louisgag ( I am not sure about the complete path )

Create a symlink of site-packages of conda into the Blender. Make sure you are still in the directory
~/Downloads/blender-2.78/2.78/python/lib/python3.5 , the directory in which you deleted the
site-packages directory.
--- For virtual environment
ln -s ~/miniconda3/envs/py35/lib/python3.5/site-packages

--- For default conda environment
ln -s ~/miniconda3/lib/python3.5/site-packages/

Now, every package you install through conda install , will be directly available for import in Blender.

@janga1997 does netcdf and plotting work for you using this method? That was quite a lot of work to find out the same old problem..

[louisgag]

PS: you can see my whole installation procedure here, perhaps you'll see a missing package... bash_output.txt

[janga1997]

@louisgag I have been using netCDF using this method.

In scripting mode, try import netCDF4, import tinycss, or any of the packages you manually installed. If the symlinks and everything are in order, they should be imported. If it shows an import error, maybe the symlink was wrong, because I see you have installed the packages in the py35 environment.

Can you check if the symlink site-packages of Blender points to the site-packages of py35 conda environment ?

[louisgag]

@janga1997 ok, found out the issue is with the hdf package, a forced upgrade is needed (at least on my Ubuntu 14.04). See the comment by jjhelmus for info: https://github.com/ContinuumIO/anaconda-issues/issues/1360#issuecomment-274627285

The fix is this: conda install hdf4=4.2.12

Now I can import netcdf results, but plotting is not working... I'm assuming cairo/jpg may have been broken by the forced downgrade cause by the hdf upgrade...

untitled.crash.txt

[louisgag]

But it seems only hdf depends on jpeg, so that may not be the issue...

[janga1997]

@louisgag If cairosvg and pygal are installed, then plotting should work. Because in base.py, we try to import pygal and .plotlib. In plotlib.py, we try to import cairosvg and pygal. If these two packages import correctly in your Scripting mode console, then you should be fine. So cairo may be the only issue.

[louisgag]

Just checked, they do import correctly

[janga1997]

@louisgag So any other issues? Does it work properly?

[louisgag]

@janga1997 Only plotting has issues ;-)

[janga1997]

Windows Everything is identical in Windows, except the symlinks and switching to an environment. After installing the Windows executable from https://conda.io/miniconda.html, go ahead and create a new environment with the exact same command.

conda create -n py35 python=3.5.2 Go ahead and install the necessary packages.

Switching to environments is different in Windows a bit It is activate py35 (instead of source activate py35) and deactivate py35

Now, if you're using a extracted Blender, go ahead and directly delete the site-packages (same path as in Ubuntu, {blender_directory}\2.78\python\lib\python3.5\site-packages)

If it is a system installation of Blender, you will likely need Administer privileges to delete a folder from a Program. In my case, Blender installed at C:\Users\janga\Program Files\Blender Foundation\2.78\python\lib\python3.5\site-packages The rest of the path is the same as the extracted one.

symlinks in windows are made through the program mklink (But in the reverse order) Open up the command prompt with the option Run as Administrator.

In Ubuntu it is ln -s /original_folder /symlink In Windows it is mklink /D \symlink \original_folder (the /D stands for directory)

In my case, Anaconda ( or miniconda ) was installed at C:\Users\janga\Anaconda\py35\lib\python3.5\site-packages