FilipDominec / python-meep-install

Robust compilation procedure of latest MEEP and python-meep on Linux
GNU General Public License v2.0
16 stars 19 forks source link

ubuntu 16.04: missing -fPIC in MPB and "Unknown SWIG preprocessor directive" #2

Closed aitatanit closed 8 years ago

aitatanit commented 8 years ago

Hi @FilipDominec, I'm having issues getting things to work on the latest ubuntu. Attached is the log file.

Thanks ubuntu-16.04.txt

FilipDominec commented 8 years ago

Hi @aitatanit , thank you for attaching the log file. So far, it appears that everything installed correctly; in fact it looks like those log files I got myself when testing the installer. What are the issues exactly?

aitatanit commented 8 years ago

Now that I look at the file again I see what you mean but the screen output is a bit misleading; Here is what I got that prompted the issue - I just wanted to make sure all installed ok even with this output:

make all-recursive make[1]: Entering directory '/home/aliean/Downloads/python-meep-install/meep' Making all in src make[2]: Entering directory '/home/aliean/Downloads/python-meep-install/meep/src' make all-am make[3]: Entering directory '/home/aliean/Downloads/python-meep-install/meep/src' CXXLD libmeep_mpi.la /usr/bin/ld: /usr/lib/libmpb.a(libmatrices_la-eigensolver.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC /usr/lib/libmpb.a: error adding symbols: Bad value collect2: error: ld returned 1 exit status Makefile:494: recipe for target 'libmeep_mpi.la' failed make[3]: * [libmeep_mpi.la] Error 1 make[3]: Leaving directory '/home/aliean/Downloads/python-meep-install/meep/src' Makefile:424: recipe for target 'all' failed make[2]: * [all] Error 2 make[2]: Leaving directory '/home/aliean/Downloads/python-meep-install/meep/src' Makefile:492: recipe for target 'all-recursive' failed make[1]: * [all-recursive] Error 1 make[1]: Leaving directory '/home/aliean/Downloads/python-meep-install/meep' Makefile:401: recipe for target 'all' failed make: * [all] Error 2 Reading package lists... Done Building dependency tree
Reading state information... Done Note, selecting 'libpython2.7-stdlib' instead of 'python-argparse' libpython2.7-stdlib is already the newest version (2.7.11-7ubuntu1). python-dev is already the newest version (2.7.11-1). python-numpy is already the newest version (1:1.11.0-1ubuntu1). python-matplotlib is already the newest version (1.5.1-1ubuntu1). python-scipy is already the newest version (0.17.0-1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Directories for SWIG -I option : ['/usr/local/include'] Directories for -L option : ['/usr/local/lib'] running clean 'build/lib.linux-x86_64-2.7' does not exist -- can't clean it 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-2.7' does not exist -- can't clean it Directories for SWIG -I option : ['/usr/local/include'] Directories for -L option : ['/usr/local/lib'] running build_ext building '_meep_mpi' extension swigging meep_mpi.i to meep_mpi_wrap.cpp swig -python -c++ -w503,389,302,362,314,509,451 -I/usr/local/include -o meep_mpi_wrap.cpp meep_mpi.i meep-site-init.py:1: Error: Unknown SWIG preprocessor directive: initialisations (if this is a block of target language code, delimit it with %{ and %}) meep_common.i:87: Error: Unable to find 'meep/mympi.hpp' meep_common.i:88: Error: Unable to find 'meep/vec.hpp' meep_common.i:89: Error: Unable to find 'meep.hpp' error: command 'swig' failed with exit status 1 Directories for SWIG -I option : ['/usr/local/include'] Directories for -L option : ['/usr/local/lib'] running install running build running build_py file meep_mpi.py (for module meep_mpi) not found file meep_mpi.py (for module meep_mpi) not found running build_ext building '_meep_mpi' extension swigging meep_mpi.i to meep_mpi_wrap.cpp swig -python -c++ -w503,389,302,362,314,509,451 -I/usr/local/include -o meep_mpi_wrap.cpp meep_mpi.i meep-site-init.py:1: Error: Unknown SWIG preprocessor directive: initialisations (if this is a block of target language code, delimit it with %{ and %}) meep_common.i:87: Error: Unable to find 'meep/mympi.hpp' meep_common.i:88: Error: Unable to find 'meep/vec.hpp' meep_common.i:89: Error: Unable to find 'meep.hpp' error: command 'swig' failed with exit status 1 Directories for SWIG -I option : ['/usr/local/include'] Directories for -L option : ['/usr/local/lib'] running bdist running bdist_dumb running build running build_py file meep_mpi.py (for module meep_mpi) not found file meep_mpi.py (for module meep_mpi) not found running build_ext building '_meep_mpi' extension swigging meep_mpi.i to meep_mpi_wrap.cpp swig -python -c++ -w503,389,302,362,314,509,451 -I/usr/local/include -o meep_mpi_wrap.cpp meep_mpi.i meep-site-init.py:1: Error: Unknown SWIG preprocessor directive: initialisations (if this is a block of target language code, delimit it with %{ and %}) meep_common.i:87: Error: Unable to find 'meep/mympi.hpp' meep_common.i:88: Error: Unable to find 'meep/vec.hpp' meep_common.i:89: Error: Unable to find 'meep.hpp' error: command 'swig' failed with exit status 1

FilipDominec commented 8 years ago

Hi, I see the direct cause of the error, i.e., the -fPIC option is needed on the 64-bit systems for proper linking of compiled libraries.

But I will have to check why my tests did not fail previously, and what actually changed between the versions. Thank you for reporting this.

aitatanit commented 8 years ago

thanks. I can't import meep or meep-mpi so I know something went wrong with the install.

aitatanit commented 8 years ago

Also, do you know if python-meep plays nice with Anaconda Python?

FilipDominec commented 8 years ago

No, I did not try using Anaconda. I guess there should be no problem once you compile MEEP and set up all paths correctly. But this is a different issue.

FilipDominec commented 8 years ago

@aitatanit , I could not reproduce the reported bug by running python-meep-install.py on freshly installed ubuntu 16.04 on amd64 virtual machine.

I have compared the logs from your system (attached above) and from the freshly installed ubuntu. There are minor differences, such as the presence of gawk, latex2html in your system.

However a more important difference between the logs may be the line 533 (which contains maxwell_target_operator1 in -lmpb... yes. It implies you have a different version of mpb-dev than that available in the up-to-date repository. Maybe you downloaded the MPB's sources and compiled it yourself? If so, you have to repeat its compilation with the -fPIC option enabled.

aitatanit commented 8 years ago

@FilipDominec , thanks for the reply. MPB was installed from the repo. I'll purge the repo installs and try installing meep and python-meep alone then try to install mpb (from source and note repo).

aitatanit commented 8 years ago

I have it working on 14.04, however, I had to manually install mpb-dev since it's not in the repo. I got it from launchpad

I still cannot make it work in 16.04. It seems to be a swig issue. Attached are two file, one with MPI and one with SERIAL options. Can you please check into this one more time? Thanks output16.04_mpi.txt output16.04_serial.txt

FilipDominec commented 8 years ago

I have, too, independently encountered the bug stating Error: Unknown SWIG preprocessor directive. This is apparently due to some changes in SWIG syntax. Unfortunately I have no means to modify the python-meep project, but I will try to issue a workaround in its installation script.

aitatanit commented 8 years ago

I'll dig into it a bit if time allows but I have a working version in 14.04 for the time being. When I run the test code in the python-meep folder I get an error for brag-transmission.py. I attached a screenshot of thr output for 3 scripts. What do you make of the error in brag? screenshot from 2016-06-06 13-05-03

FilipDominec commented 8 years ago

First, the test failure should be put into a separate issue than the original report (which I am thankful for and I have issued a fix a minute ago).

Second, it is apparently not related to the python-meep-install project whatsoever.

Third, I do not have a clue what might go wrong. Thank you, however for your cooperation on tracing the aforementioned bug. Regards, Filip

aitatanit commented 8 years ago

@FilipDominec, I just pulled from your latest repo and the code compiled on Ubuntu 16.04 just fine. To make the test cases run in python-meep folder I had to change import meep to import meep_mpi but they worked fine from that point on. Even the brag code worked this time. Thanks a bunch.

FilipDominec commented 8 years ago

Recently, I was also unable to compile MEEP with the MPB support, apparently due to the "-fPIC" flag unset during the compilation of MPB in the Ubuntu repository for the 64-bit system.

The correct solution will be to recompile own copy of MPB. For now, I disabled the support of MPB-related functions in MEEP by uninstalling the 'mpb-dev' package.

aitatanit commented 8 years ago

How about using the MPB package from the repo?

FilipDominec commented 8 years ago

@aitatanit As I wrote above, I used the mpb-dev package from the repo, but this is the cause of the error.

i2000s commented 7 years ago

@aitatanit @FilipDominec Could you provide a working script for installation? I was following @FilipDominec 's script for Ubuntu 14.04, but it didn't work on Ubuntu 16.04. Similar error to OP.

i2000s commented 7 years ago

@FilipDominec I think meep with openmpi works now with the latest script in this repo. I am not sure what do you mean on the -fPIC parameter to install MPB.

FilipDominec commented 7 years ago

@i2000s, your previous two posts seem a bit cotradictory, so I am a bit unsure whether you can or cannot compile meep using the script. I can test out the compilation on my system, if you wish.

The -fPIC parameter is needed on 64-bit systems for a library that is to be used by another program (here it is mpb and meep-mpi, respectively).

i2000s commented 7 years ago

@FilipDominec Thank you for replying. I have used your script to install meep-openmpi, and it seems working except for all tests failed because from meep import * is not correct. But when I went through your script, I found the mpb package was commented out (not installed). I am confused about your decision to not support MPB. Is that as simple as sudo apt install -fPIC mpb-dev or sudo apt install mpb h5utils qtiplot nlpot? Or, I should download MPB source code and do the ./configure -fPIC && make && make install? Or, can I copy the same script for libctl for MPB compiling? I didn't find this -fPIC switch on the MPB website. A helpful reference is highly appreciated. Thank you for making the script!

FilipDominec commented 7 years ago

@i2000s I think that the apt installer has nothing to do with the -fPIC option, which has to be specified at compile time instead. You will have to compile MPB yourself with -fPIC as you suggest. It could have been included in the scripts, but I was too lazy to discover it and to do all tests over again.

Unfortunately, I have no experience compiling MPB and I am not going to use it in the close future. If you are successful in this task, I will be very happy to add your compilation commands to this project.

i2000s commented 7 years ago

Let me tried the MPB compilation later. But for now, there is still a problem with MEEP which I hope you know how to fix. I just installed MEEP with openmpi using your script on a new computer with Ubuntu 16.04 amd64 installed. But in the test codes, if I try

import meep_openmpi as meep
import meep

or replace meep_openmpi with meep_mpi, I will get an error saying ImportError: No module named 'meep_openmpi'. which meep-mpi will give /usr/local/bin/meep-mpi on my PATH. Any thought on this?

Sorry for asking tangential questions here.

aitatanit commented 7 years ago

I have it working on 14.04, let me try it on 16.04 and see what comes up.

aitatanit commented 7 years ago

Just installed it on 16.04 and from ipython I got this, which seems to be working. Did you install all the python libraries? Do you have the installation script's output? I cloned the repo and started from scratch, btw.


$ ipython
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
Type "copyright", "credits" or "license" for more information.

IPython 2.4.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import meep_mpi
Python-meep starting...
Using MPI version 3.0, 1 processes
Global initialisation done.
Site-specific initialisation done.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<`
>>> Welcome to Python-Meep version 1.4.2 - running in MPI mode. <<<<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

In [2]: 
i2000s commented 7 years ago

Maybe because I have named Python 2.7.11 as python2 and Python 3.5.1 as python. But does this matter for importing meep_mpi? So, I just tried redo the installation process on top of what I have installed. Then I got errors as attached... I think the first installation was success, because I could run meep-mpi in the terminal. BTW, my computer crashed this morning and I have to upgrade my kernel to 4.7.0 but I don't feel that will affect MEEP. Hopefully you can help find some idea in the log. Thanks. log.txt

aitatanit commented 7 years ago

The python version maybe the problem since I see that the installation is wanting to work with version 2.7 (see lines 3522 & 3526 in your log file). I suggest starting fresh with python -> python 2.7 and see if this is actually the issue.

FilipDominec commented 7 years ago

Hi, as a matter of fact, I did not test the compilation under python 3 so far. It may need some hacking.

Dne středa 23. listopadu 2016 Jimmy Touma notifications@github.com napsal(a):

The python version maybe the problem since I see that the installation is wanting to work with version 2.7 (see lines 3522 & 3526 in your log file). I suggest starting fresh with python -> python 2.7 and see if this is actually the issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/2#issuecomment-262509522, or mute the thread https://github.com/notifications/unsubscribe-auth/AAfPzoeDox3qoUMnhNF97zmnIcZ2QmcTks5rBDxCgaJpZM4IuC1d .

i2000s commented 7 years ago

I tried using python 2.7 on the installation, but no luck either... I didn't find any obvious error. Attached is the log. Thanks. log.txt

aitatanit commented 7 years ago

puzzling ..

Are you using Anaconda python or Enthought Canopy as your python base or are you using the packages that come with Ubuntu?

aitatanit commented 7 years ago

Here is my installed python packages (from the Ubuntu 16.04 repos), maybe if you can compare my python 2.7 packages with yours something will pop-up to fix your issue: installed_python.txt

aitatanit commented 7 years ago

Here are the installed python packages from another computer where python meep is working installed_python2.txt

i2000s commented 7 years ago

Yeah, I forgot to mention, I indeed used Anaconda python. Let me look at your log.

aitatanit commented 7 years ago

Okay, that is the problem that I had. I change between the two and lately I don't use Anaconda anymore, since all the packages that I need are in the Ubuntu repos. Comment out the line that Anaconda put in your .bashrc, launch a new terminal and try the installation again. It should work this time.

i2000s commented 7 years ago

@aitatanit This does seem fixing the installation problem. Thank you! But when I run

import meep_mpi as meep
import meep

it gave me an error "no module is named meep". I did the test over the 2D_convergence.py code. If I just comment out the import meep code, then the error becomes name *quiet* is not defined. Being strange again...

aitatanit commented 7 years ago

Don't use the 2nd import, just use

import meep_mpi as meep
aitatanit commented 7 years ago

I only tested a couple of the included examples and they worked after I replaced

import meep

with

import meep_mpi as meep
i2000s commented 7 years ago

Ok, I found a working way to do this:

from meep_mpi import *
import meep_mpi as meep

that works for the case MPI is enabled. Changing the order of the two lines doesn't work. Without the first line doesn't work. Hooray! Thank you for being around!

aitatanit commented 7 years ago

glad you got it working, happy meeping

FilipDominec commented 7 years ago

It appears somewhat puzzling to me that 'changing the order of the two lines' causes an error. How this is possible? I would be thankful if you pasted the typical error message here.

Btw. I would recommend to keep all meep functions in their own namespace for the sake of clarity. But this is just a matter of style.

i2000s commented 7 years ago

Hah, you are more careful than me. If I change the order of the two lines, it also works perfectly. My mistake on this.

Could you illustrate a little more on 'keep all meep functions in their own namespace'?

FilipDominec commented 7 years ago

By the last suggestion, I only meant that the code remains easier to read (for me) if I use import meep; meep.x instead of from meep import *; x. But it is only a matter of style.

i2000s commented 7 years ago

Those are the things I need to learn. Thank you for the suggestion on styling.