Guile (2.0.13) is incompatible with meep: Temporary workaround

[JohnWeiner]

It appears that the latest version of Guile (2.0.13) is incompatible with meep. The first meep tutorial (straight waveguide) does not run with Guile 2.0.13 installed. Here is a summary of what I did to get it to work (in single processor mode)

1. Remove Guile 2.0.13 using apt remove
2. Download deb packages for Guile 2.0.11, Guile-dev 2.0.11, Guile-libs 2.0.11
3. Use dpkg -i for installation in the following sequence: (a) Guile-lib, (b) Guile, (c) Guile-dev
4. Use the python-meep-install.sh script, but edit it so as to choose the “serial” option and comment out the line that invokes installation of guile.

I found that the install script runs to completion. Even though the meep mode is “serial” (single processor, not multiprocessor) the command to run meep is still meep-mpi. A check of the guile version shows 2.0.11 installed.

The first tutorial on the Meep tutorial web page (straight wave guide) ran without error and generated correct h5 files. The uses h5utils to convert the h5 files to .png files ran without error.

[FilipDominec]

Thank you, John. This appears as a fundamental incompatibility which should be fixed upstream, i.e. by actual MEEP developers, and they should be aware of it now.

If nothing changes and MEEP does not compile in new Ubuntu until September 2017, I will take some measures to explicitly download the legacy libraries.

[JohnWeiner]

Hi Filip,

I think that Ardavan is aware of the problem…he seems to think it has to do with SWIG that has failed to take account of a change that Guile made between versions 2.0.11 and 2.0.12.

In fact the installation of meep-1.3 is not too difficult on the Ubuntu box. The meep package does have all the dependencies, but I don’t think there is any way to tell the package handler, apt or apt-get, to install guile-2.0.11 instead of the latest version.

Here is essentially what I did to get around the problem:

First, I installed guile-2.0.11 on /usr/local from downloading the tarball and building it from sources. I will send the list of dependencies I needed if and when you want them, but I think most, if not all, of them are already in the meep package. I then modified the $PATH to make sure that /usr/local/bin is searched BEFORE /usr/bin. It might be that a more recent guile version exists (either from the Ubuntu distro or the meep package) in /usr/bin, /usr/lib, /usr/include, but if the search path starts with /usr/local/bin, then the guile-2.0.11 will be invoked.

Second, there is a real bug with a file called stime.h that is in …/libguile. The version that comes with guile 2.0.11 does not work, but all you have to do is copy stime.h from a later version into the libguile that is installed at the /usr/local level.

With those two fixes everything else should be ok on Ubuntu.

On the macOS I had a lot problems because the dependencies are not well-documented and one just finds what is missing by trial and error. Now everything is running well on both systems and I can get back to substantive, scientific problems.

Best regards,

John

On 1 Aug 2017, at 22:47, Filip Dominec notifications@github.com wrote:

Thank you, John, and thank you again for posting more elaborate guide in the mailing list http://meep-discuss.ab-initio.mit.narkive.com/3sHyPdvB/notes-on-installing-guile-meep-on-macos-sierra-version-3-july-2017. This appears as a fundamental incompatibility which should be fixed upstream, i.e. by actual MEEP developers, and they should be aware of it now.

If nothing changes and MEEP does not compile in new Ubuntu until September 2017, I will take some measures to explicitly download the legacy libraries.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-319492041, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6eBDnx2rtepLMoUzp5uSn9IeBmV3qks5sT47fgaJpZM4N9eXD.

[FilipDominec]

John (and all folks bitten by the update of Guile and obsoleteness of Swig), as I promised, I added a workaround that detects that package Guile1.8 is not available, and that Guile2.0 is also not available in its version of 2.0.11 or older.

In such a case, it downloads the Guile git source, and checkouts to its old-enough version. Thus, the meep compilation seems to be fine on Ubuntu 17.04, as suggested by the latest log https://github.com/FilipDominec/python-meep-install/blob/master/logs/171006-ubuntu17.04-amd64-02withguile11fromgit.log

Note that there is another compatibility issue in installation on new Ubuntu versions, as reported here: https://github.com/FilipDominec/python-meep-install/issues/10

I hope that the described update spares you some time.

[FilipDominec]

As I think about it - perhaps your approach of cleanly installing Guile into the /usr/local/bin is preferable. Could you please post the code that does the job here?

So far, I probably did not encounter the bug with stime.h. Could you please describe the situation when it becomes a trouble?

[JohnWeiner]

Hi Filip,

Thanks for the new script. I browsed the log file looking at the various warnings. Some of them look familiar, but as long as warnings don’t turn into errors the installation continues. I did note on the very last line of the log file,

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1.

Is this as ominous as it sounds?

I haven’t had the courage to try python-meep or mpi-meep yet. I have a working scheme-meep system that runs on a single core. For long parameter sweeps, it takes longer than a parallel version would of course, but at least I’ve been able to work on some physics questions and take a break from system/development problems. My next installation project will be to get mpi-meep working. I have a motherboard with 16 cores in the cpu. My hope is that they will work in parallel once the mpi version is installed.

With respect to installing guile-2.0.11 in /usr/local/bin, I just downloaded the pertinent tarball from the gnu site and went through the usual ./config make make install sequence.

One has to pay careful attention to dependencies of course. I think that either the README or INSTALL files that come with the tarball has a list of the needed dependencies. My experience has been that using the package handlers like home-brew, MacPorts, or apt leads to problems. They download and install the latest versions of everything, and sometimes that leads to problems with guile-2.0.11. All of this will go away, I presume, once swig is updated…don’t know who maintains swig or even if they are aware of the problem.

Best regards,

John

On 5 Oct 2017, at 23:51, Filip Dominec notifications@github.com wrote:

—

[FilipDominec]

Hi, John, thanks for your message. As you probably assume,

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1.

is actually fatal for python-meep compilation. It however means that something failed in the whole compilation chain, and one has to find out what. I spent about two hours with this investigation, and eventually I found out that the recent git version of the script compiles correctly under Ubuntu 17.04 (amd64) with Guile 2.0.13 taken from the repository. This is puzzling, since I thought this is exactly the version (both of Guile and of Ubuntu) when the compilation broke.

Regards, Filip

[JohnWeiner]

Hi Filip

It used to be that all versions of Guile later than 2.0.11 would not RUN correctly, even when they compiled and linked to meep correctly. What I had found out was that, when running the first waveguide tutorial, I would get a run time error that failed to produce usable output. Furthermore, the h5 utils, especially h5topng, threw segmentation faults.

Ardavan Oskooi surmised that it had to do with SWIG not adjusting to a Guile change after 2.0.11, but I have no independent judgment about that.

Maybe SWIG has caught up with Guile. There have been a few messages, one from SJ, on the meep-discuss about later versions of guile now working, but other messages contradicting this good news. I don’t know what the status of things is right now. I do know that I have a working meep environment using guile-2.0.11 running on an iMac, a MacBook, and Ubuntu 17.04 (amd64).

As far as I know Guile-2.0.11 always works (for the time being), although some aspects of it have apparently been deprecated and may eventually disappear.

Best regards,

John

On 8 Oct 2017, at 23:26, Filip Dominec notifications@github.com wrote:

Hi, John, thanks for your message. As you probably assume,

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1.

is actually fatal for python-meep compilation. It however means that something failed in the whole compilation chain, and one has to find out what. I spent about two hours with this investigation, and eventually I found out that the recent git version of the script compiles correctly under Ubuntu 17.04 (amd64) with Guile 2.0.13 taken from the repository. This is puzzling, since I thought this is exactly the version (both of Guile and of Ubuntu) when the compilation broke.

Regards, Filip

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-335039767, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6ePPBsG00-jQ4qapfJ7aSFN5_CpS0ks5sqT3pgaJpZM4N9eXD.

[FilipDominec]

Hi, I can positively state that the installer script and both the simulations finish successfully with Guile 2.0.13 installed from the Ubuntu-17.04 repository. So it seems that the original issue somehow magically disappeared.

[JohnWeiner]

Try it with Guile 2.2.2…I think that is the latest.

John

On 9 Oct 2017, at 10:02 , Filip Dominec notifications@github.com wrote:

Hi, I can positively state that the installer script and both the simulations finish successfully with Guile 2.0.13 installed from the Ubuntu-17.04 repository. So it seems that the original issue somehow magically disappeared.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-335088628, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6eJG61aINnfr7Lfvu7rpUJJWDw9KEks5sqdMTgaJpZM4N9eXD.

[JohnWeiner]

Hi again Filip

I think guile 2.2 is the version packaged with homebrew and apt-get. If it works with 2.2 then meep can be installed from the binary packages which simplifies life for everybody.

John

Envoyé de mon iPhone

Le 9 oct. 2017 à 10:02, Filip Dominec notifications@github.com a écrit :

Hi, I can positively state that the installer script and both the simulations finish successfully with Guile 2.0.13 installed from the Ubuntu-17.04 repository. So it seems that the original issue somehow magically disappeared.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[FilipDominec]

Hi, as I already mentioned in my recent post under #10, I made a new test over different Ubuntu versions. In short, the script finishes correctly in all of them without the need to manually compile Guile. I do not know what changed, but it just works.

Here I post the versions of Guile found in each system.

Ubuntu version     "guile-1.8-dev"      "guile-2.0-dev"      Compilation result
--------------     ---------------      ---------------      ------------------
16.04 "trusty"     1.8.8+1-10ubuntu1    2.0.11+1-10          OK with 1.8
16.04.3 "trusty"   1.8.8+1-10ubuntu1    2.0.11+1-10          OK with 1.8
16.10 "yakkety"    1.8.8+1-10ubuntu1    2.0.11+1-12build2    OK (both with 1.8 and 2.0)
17.04 "zesty"      (N/A)                2.0.13+1-4           OK (now) with 2.0
17.10 "artful"     (N/A)                2.0.13+1-4           OK (as in "zesty")

Note that no package like "guile-2.2-dev" exists in any official repo (yet).

[JohnWeiner]

Filip,

Is there some way I can try your script without endangering the working installation I already have? I think you mentioned once something about a “virtual machine” that could be used to test an installation independently of an installation already in place. I don’t know anything about virtual machines and would be grateful for good links where I could learn about them. If everything works with the apt-get package installer “as is” so much the better. But I would like to verify it on my Ubuntu box and on macOS.

Best regards,

John

On Oct 10, 2017, at 08:42, Filip Dominec notifications@github.com wrote:

Hi, as I already mentioned in my recent post under #10 https://github.com/FilipDominec/python-meep-install/issues/10, I made a new test over different Ubuntu versions. In short, the script finishes correctly in all of them without the need to manually compile Guile. I do not know what changed, but it just works.

Here I post the versions of Guile found in each system.

Ubuntu version "guile-1.8-dev" "guile-2.0-dev" Note 16.04 "trusty" 1.8.8+1-10ubuntu1 2.0.11+1-10 Fine. 16.04.3 "trusty" 1.8.8+1-10ubuntu1 2.0.11+1-10 Fine 16.10 "yakkety" 1.8.8+1-10ubuntu1 2.0.11+1-12build2 Apt-get install 2.0 and everything is fine. 17.04 "zesty" (N/A) 2.0.13+1-4 Apt-get install 2.0 - now it seems to be fine, too, see logs/*GuileFromRepo.log 17.10 "artful" (N/A) (2.0.13+1-4) Same version as in "zesty", and the same good result.

Note that no package like "guile-2.2-dev" exists in any official repo (yet).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-335459545, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6eGXnSYlwH9ZtRW-Va38AVpNDRDmBks5sq2YcgaJpZM4N9eXD.

[FilipDominec]

On Linux, I use the following procedure:

1. get the image, e.g. wget http://releases.ubuntu.com/16.04/ubuntu-16.04.3-desktop-amd64.iso
2. qemu-img create ubuntu-16.04.3-desktop-amd64.img 10G
3. run the virtual machine qemu-system-x86_64 -m 1000 -smp cpus=2 -enable-kvm -hda ubuntu-16.04.3-desktop-amd64.img -cdrom ubuntu-16.04.3-desktop-amd64.iso
4. install ubuntu to the virtual disk image (this can not harm any other file on your disk), shut down the virtual machine
5. re-run the machine without cdrom qemu-system-x86_64 -m 1000 -smp cpus=2 -enable-kvm -hda ubuntu-16.04.3-desktop-amd64.img
6. once booted, launch a terminal (e.g. xterm) window and get the fresh installer from git sudo apt-get install git -y ; git clone http://github.com/filipdominec/python-meep-install --depth 1 ; cd python-meep-install
7. follow the command under Testing and debugging in the project's README.md file, that is: sudo ./python-meep-install.sh 2>&1 | tee logs/date +%y%m%d-lsbrelease -sd | tr ' ' '' -uname -m.log
8. after ca. 10 minutes, it should end up successfully and you can run either meep from console, or launch python and test the python-meep module by import meep_mpi

[JohnWeiner]

Hi Filip and thanks for this procedure. I am traveling in the States for the rest of the month; but when I return to Paris, I’ll install qemu then attempt the virtual machine on my Linux/Ubuntu box. I presume I could also experiment with simply downloading the apt-get meep package from the Ubuntu repository and see if it installs and runs correctly on the virtual machine.

Best regards,

John

Le mar. 10 oct. 2017 à 10:26, Filip Dominec notifications@github.com a écrit :

On Linux, I use the following procedure:

1. get the image, e.g. http://releases.ubuntu.com/16.04/ubuntu-16.04.3-desktop-amd64.iso
2. `qemu-img create ubuntu-16.04.3-desktop-amd64.img 10G
3. run the virtual machine qemu-system-x86_64 -m 1000 -smp cpus=2 -enable-kvm -hda ubuntu-16.04.3-desktop-amd64.img -cdrom ubuntu-16.04.3-desktop-amd64.iso
4. install ubuntu to the virtual disk image (this can not harm any other file on your disk), shut down the virtual machine
5. re-run the machine without cdrom qemu-system-x86_64 -m 1000 -smp cpus=2 -enable-kvm -hda ubuntu-16.04.3-desktop-amd64.img
6. once booted, sudo apt-get install git -y ; git clone http://github.com/filipdominec/python-meep-install ; cd python-meep-install
7. follow the command under Testing and debugging in the project's README.md file, that is: sudo ./python-meep-install.sh 2>&1 | tee logs/date +%y%m%d-lsbrelease -sd | tr ' ' '' -uname -m.log
8. after ca. 10 minutes, it should end up successfully and you can run either meep from console, or launch python and test the python-meep module by import meep_mpi

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-335547664, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6eCsr6_0o9I0VFN0rFNq7FhptH-Ofks5sq6i1gaJpZM4N9eXD .

-- John Weiner 22 Avenue de la Sibelle 75014 Paris France Tel: (mobile) 33 (0)6 87 93 43 03; Tel: (fix) 33 (0)9 67 46 87 22

[JohnWeiner]

Hi Filip,

Returning to Paris yesterday I found that my GRUB boot program on my Ubuntu box was broken and would not boot the system. Finally was forced to repair GRUB2, and in the process the old 17.04 (with a partitioned disk) was lost. So now I have a new 17,04 up and running cleanly, and I would like to install python-meep (preferably with mpi) on this fresh operating system. I just verified that the “sudo apt-get install meep” still loads guile-2.0.13 and it still does work with always the same message at meep run time. On your Git-Hub repository you have some notes about the guile-2.0.13 problem for 17.04 and a “fix underway”.

Can you bring me up to date on “the fix”? Also, could I just edit the python-meep script somewhere to invoke guile-2.0.11 instead of the default 2.0.13?

Best regards,

John

[FilipDominec]

Dear John, I fixed the line in the README.md file. Actually, in my last tests, also your current Ubuntu version worked well with guile-2.0.13 which we know to have caused some troubles previously.

I thus suggest you may try to install meep exactly using the python-meep-install.sh script. Please make sure you keep the log file

• if there is anything wrong with your installation, I am going to issue a fix soon.

Filip

2017-11-03 13:14 GMT+01:00, John Weiner notifications@github.com:

Hi Filip,

Returning to Paris yesterday I found that my GRUB boot program on my Ubuntu box was broken and would not boot the system. Finally was forced to repair GRUB2, and in the process the old 17.04 (with a partitioned disk) was lost. So now I have a new 17,04 up and running cleanly, and I would like to install python-meep (preferably with mpi) on this fresh operating system. I just verified that the “sudo apt-get install meep” still loads guile-2.0.13 and it still does work with always the same message at meep run time. On your Git-Hub repository you have some notes about the guile-2.0.13 problem for 17.04 and a “fix underway”.

Can you bring me up to date on “the fix”? Also, could I just edit the python-meep script somewhere to invoke guile-2.0.11 instead of the default 2.0.13?

Best regards,

John

-- You are receiving this because you were assigned. Reply to this email directly or view it on GitHub: https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-341687442

[JohnWeiner]

Hi Filip,

I noticed in my last message to you I left out the word “not” in a critical sentence. It should have read, "I

just verified that the “sudo apt-get install meep” still loads guile-2.0.13 and it still does NOT work, with always the same error message, at meep run time.

So, the meep package available at the Ubuntu repository still has problems…that is what I meant to say.

Looking back through the meep-discuss mail archive, I discovered a message from Ardavan Oskooi, “Installation instruction for Meep on Ubuntu 16.04”, dated 26 July 2017. The instructions are a detailed step-by-step installation of all dependencies and meep from sources for guile-meep and guile-meep-mpi. Ardavan references the problem with SWIG in his meep-discuss message, and in the instructions he specifies the installation of guile-2.0.11 rather than 2.0.13 or later versions. Being under time pressure to get things up and running on the Ubuntu box, I decided to give the instructions a try on Ubuntu 17.04. As of this morning, meep and meep-mpi both appear to be running without error. I haven’t yet tested h5utils, but I am guardedly optimistic that these ‘from sources’ instructions are valid for 17.04 as well as 16.04.

So, observing the old Yankee adage, “If it ain’t broke, don’t fix it”, I have not yet tried your script since things seem to be working now; and I have to get some substantive work done before a deadline at the end of the month. I am reassured to know that your python-meep-install.sh script should work for Ubuntu 17.04 as well, but I am still unclear why guile-2.0.13 works for this script but does not work in the Ubuntu meep package. If I run into problems with the present installation, I will try to re-install with your script, which would give me the python option as well.

Best regards,

John

On 4 Nov 2017, at 20:31, Filip Dominec notifications@github.com wrote:

Dear John, I fixed the line in the README.md file. Actually, in my last tests, also your current Ubuntu version worked well with guile-2.0.13 which we know to have caused some troubles previously.

I thus suggest you may try to install meep exactly using the python-meep-install.sh script. Please make sure you keep the log file

• if there is anything wrong with your installation, I am going to issue a fix soon.

Filip

2017-11-03 13:14 GMT+01:00, John Weiner notifications@github.com:

Hi Filip,

Returning to Paris yesterday I found that my GRUB boot program on my Ubuntu box was broken and would not boot the system. Finally was forced to repair GRUB2, and in the process the old 17.04 (with a partitioned disk) was lost. So now I have a new 17,04 up and running cleanly, and I would like to install python-meep (preferably with mpi) on this fresh operating system. I just verified that the “sudo apt-get install meep” still loads guile-2.0.13 and it still does work with always the same message at meep run time. On your Git-Hub repository you have some notes about the guile-2.0.13 problem for 17.04 and a “fix underway”.

Can you bring me up to date on “the fix”? Also, could I just edit the python-meep script somewhere to invoke guile-2.0.11 instead of the default 2.0.13?

Best regards,

John

-- You are receiving this because you were assigned. Reply to this email directly or view it on GitHub: https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-341687442 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-341923835, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6eKtXkV6777OE2EFjj8FcNTqt0o3Pks5szLuSgaJpZM4N9eXD.

[FilipDominec]

Hi, this is really puzzling, since I tested my script successfully also on 17.04 without any error in using python-meep. What can be the difference of your setup and the default installation of Ubuntu 17.04 (amd64) I have tested lately? I have one idea - did you install latest updates on your system?

[JohnWeiner]

I don’t have a good answer. Ardavan traces the problem to a lack of guile maintenance at SWIG. It might be that the Ubuntu meep package in their repository is using an older version of SWIG and that your script is using a more up-to-date version that has taken into account the changes in guile after 2.0.11. I filed a bug report with the package maintainers at Ubuntu to see if they would look into it…so far nothing has happened.

My version of 17.04 was up to date at the time I tried to install meep with “sudo apt-get install meep”. It DOES install without error, BUT at meep run time the error message comes up about something being out of position, and the code does not execute to completion. I don’t know what the ice nine trace back error messages mean that are generated when the code bombs. Understanding those messages may provide a clue to where the code goes off the rails, but now that I have a working meep environment from sources compilation and linking, I’m just using it to advance the real work I need to get done.

Best regards,

John

On 10 Nov 2017, at 17:44, Filip Dominec notifications@github.com wrote:

Hi, this is really puzzling, since I tested my script successfully also on 17.04 without any error in using python-meep. What can be the difference of your setup and the default installation of Ubuntu 17.04 (amd64) I have tested lately? I have one idea - did you install latest updates on your system?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FilipDominec/python-meep-install/issues/7#issuecomment-343524375, or mute the thread https://github.com/notifications/unsubscribe-auth/AHg6eG2cPqbGaMXWEdzsQmnjI80I8KU8ks5s1H1vgaJpZM4N9eXD.

[FilipDominec]

I understand and fully acknowledge that you are now using MEEP for science and engineering and have not much time to trace back the obsolescence of guile API. Actually I also need to focus on research now, instead of repeated testing of my installation procedures...

The good news is that Ardavan's script seems to be reliable and also cleaner than the installation of guile from repository, so I am tempted to adapt the installer to his approach. I will test it thoroughly. Thank you for your collaboration.