MUSCLE testing issue on the Eagle supercomputer

[djgroen]

I managed to install MUSCLE3 by using pip3 on the worker mode, and I am now running a simple distributed demonstration script:

~/.local/bin/muscle_manager reaction_diffusion.ymmsl & python3 ./diffusion.py --muscle-instance=macro & python3 ./reaction.py --muscle-instance=micro & wait

Locally this script works as intended, but on Eagle I get the following error:

python/3.5.1 load complete.
gmp/5.1.3 load complete.
mpfr/3.1.2 load complete.
libmpc/1.0.1 load complete.
gcc/4.8.2-14 load complete.
openmpi/1.8.1-1_gcc482 load complete.
Traceback (most recent call last):
  File "model.py", line 3, in <module>
    from libmuscle import Instance, Message
  File "/home/users/derek/.local/lib/python3.5/site-packages/libmuscle/__init__.py", line 4, in <module>
    from libmuscle.communicator import Message
  File "/home/users/derek/.local/lib/python3.5/site-packages/libmuscle/communicator.py", line 4, in <module>
    from ymmsl import Conduit, Identifier, Operator, Reference, Settings
  File "/home/users/derek/.local/lib/python3.5/site-packages/ymmsl/__init__.py", line 1, in <module>
    import yatiml
  File "/home/users/derek/.local/lib/python3.5/site-packages/yatiml/__init__.py", line 16, in <module>
    from yatiml.loader import Loader, add_to_loader, set_document_type
  File "/home/users/derek/.local/lib/python3.5/site-packages/yatiml/loader.py", line 5, in <module>
    from typing import Any, List, Type
ImportError: cannot import name 'Type'

I thought I had resolved this by doing pip3 install typing --user which succeeded, but unfortunately the exact same problem persisted. Do you perhaps know how we could tackle this?

And have you tried installing MUSCLE3 on the Eagle supercomputer yet? It's the most widely used machine in VECMA, so it may be good to try and sort this out.

If you like, I can also ask PSNC support about this directly.

[LourensVeen]

Hmm, this comment suggests that it needs at least Python 3.5.2 to work. On the other hand, YAtiML (which is what's crashing here) passes its test suite just fine on Travis on Python 3.4.8 (and also on 3.5.6).

For Python 3.4, I do have an extra line in the setup.py which installs 'typing>=3.6; python_version<"3.5"'. Which version of typing do you have installed now? Can you try to explicitly install at least 3.6?

Another thing that may help is to install the mypy package, but that's a stab in the dark.

[djgroen]

@arabnejad proposed to use virtualenv to resolve this issue, so he's looking into this now.

[LourensVeen]

Virtualenv is always a good idea; a newer Python in a miniconda environment will probably also fix it, but that's more effort and disk space than I would like to require. I'm hoping that changing that 'python_version<"3.5"' to 'python_version<"3.5.2"' will fix this.

[arabnejad]

Hi @LourensVeen , with both python V3.5 and 3.7.3 I got the same Error on VirtualEnv :(

from typing import Any, List, Type

[LourensVeen]

That's weird. Can you do a pip3 freeze and/ or pip3 list?

[arabnejad]

@LourensVeen here is the ouput

pip3 freeze
muscle3==0.1.0
typing==3.7.4.1

[arabnejad]

ok, the previous problem is solved, now, I am getting this error

muscle_manager reaction_diffusion.ymmsl & python3 ./diffusion.py --muscle-instance=macro & python3 ./reaction.py --muscle-instance=micro & wait
./reaction.py --muscle-instance=micro & waity --muscle-instance=macro & python3  
[1] 19656
[2] 19657
[3] 19658
Traceback (most recent call last):
  File "./reaction.py", line 3, in <module>
    from libmuscle import Instance, Message
ImportError: No module named 'libmuscle'
Traceback (most recent call last):
  File "./diffusion.py", line 6, in <module>
    from libmuscle import Instance, Message
ImportError: No module named 'libmuscle'
e0012:9000

@LourensVeen libmuscle is different module ??? should not be installed as part of muscle3

[LourensVeen]

Are you sure you have the virtualenv activated? pip install muscle3 should definitely make the libmuscle module available.

[arabnejad]

yes, otherwise, the muscle_manager command could not be executed

(VirtualEnv) [HiDALGO-hamid] ~ $ pip freeze
pip freeze
...
muscle3==0.1.0
...
typing==3.7.4.1
...

[LourensVeen]

That is weird. The muscle_manager executable also imports libmuscle, so apparently there it's working. Does it help if you start them in separate commands, i.e.

$ muscle_manager reaction_diffusion.ymmsl & $ python3 ./diffusion.py --muscle-instance=macro & $ python3 ./reaction.py --muscle-instance=micro & $ wait

[arabnejad]

@LourensVeen , same issue :(

(VirtualEnv) [HiDALGO-hamid] ~/muscle3_tests $ python3 ./diffusion.py --muscle-instance=macro &
  File "./diffusion.py", line 6, in <module>
    from libmuscle import Instance, Message
ImportError: No module named 'libmuscle'

[LourensVeen]

Hmm. Does which python3 give you the python3 executable in the virtualenv?

I need to get home, so I'll be back in an hour or so.

[djgroen]

I think this issue may happen because we are forced to do the pip3 install with the --user flag.

Perhaps it's worthwhile to clarify the documentation about which paths need to be set when installing in userspace? This is important, because no normal user will ever be able to do pip3 install muscle3 on a supercomputer...

[LourensVeen]

Why not? It works fine in a virtualenv or a Conda environment. I'm not sure exactly how you're setting things up, but on DAS-5, the following works fine:

user@fs0:~$ python3 -m venv ./muscle3_venv
user@fs0:~$ . ./muscle3_venv/bin/activate
(muscle3_venv) user@fs0:~$ pip3 install muscle3
(muscle3_venv) user@fs0:~$ python3
Python 3.4.5 (default, Dec 11 2017, 14:22:24) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libmuscle
>>> 

To be clear, I haven't actually tested on a cluster yet, and I'm aware of at least one issue that will prevent a simulation with instances on different nodes from working, so you're kind of on the cutting edge here. The fix for that is pretty close to the top of the to-do list (already works in C++, but I need to back-port it to Python), so hopefully later this week it'll be fixed on the develop branch at least.

[djgroen]

Hi Lourens,

I tried to manually start muscle_manager from virtualenv on Eagle, but I get the following error:

(VirtualEnv) derek@e0039:~$ muscle_manager
muscle_manager
Traceback (most recent call last):
  File "/home/users/derek/VirtualEnv/bin/muscle_manager", line 11, in <module>
    load_entry_point('muscle3==0.1.0', 'console_scripts', 'muscle_manager')()
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/muscle3-0.1.0-py3.5.egg/muscle_manager/muscle_manager.py", line 2, in <module>
    import ymmsl
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/ymmsl-0.9.1-py3.5.egg/ymmsl/__init__.py", line 1, in <module>
    import yatiml
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/yatiml-0.4.2-py3.5.egg/yatiml/__init__.py", line 16, in <module>
    from yatiml.loader import Loader, add_to_loader, set_document_type
  File "/home/users/derek/VirtualEnv/lib/python3.5/site-packages/yatiml-0.4.2-py3.5.egg/yatiml/loader.py", line 5, in <module>
    from typing import Any, List, Type
ImportError: cannot import name 'Type'
(VirtualEnv) derek@e0039:~$ 

When I check the modules however, I do see that typing is installed:

(VirtualEnv) derek@e0039:~$ pip3 freeze
pip3 freeze
atomicwrites==1.3.0
attrs==19.1.0
Click==7.0
cycler==0.10.0
grpcio==1.17.1
importlib-metadata==0.22
kiwisolver==1.1.0
more-itertools==7.2.0
mpi4py==3.0.2
msgpack==0.6.1
muscle3==0.1.0
numpy==1.16.4
packaging==19.1
pandas==0.25.1
pathlib2==2.3.4
pluggy==0.13.0
prompt-toolkit==2.0.9
protobuf==3.9.1
py==1.8.0
pyparsing==2.4.2
pytest==5.1.2
python-dateutil==2.8.0
pytz==2019.2
pyzmq==18.0.2
QCGPilotManager==0.4.1
ruamel.yaml==0.15.64
scipy==1.3.1
six==1.12.0
typing==3.7.4.1
wcwidth==0.1.7
yatiml==0.4.2
ymmsl==0.9.1
zipp==0.6.0
zmq==0.0.0

I also am able to 'find' the libmuscle module, so I don't have the issue Hamid has. But there does seem to be a feature in the typing library that MUSCLE3 uses that is not supported in the version that we have on Eagle. Could that indeed be the case?

[djgroen]

Oh, and I also tested it on the same Eagle worker node without virtualenv, and I then get exactly the same error.

So the bottleneck seems to be an issue with how the typing library is used (perhaps you're assuming a newer version than the one provided)?

[djgroen]

Okay, I streamlined some code, and now tried to install MUSCLE via script. I'm also using the HiDALGO account now, because the VECMA allocation is temporarily out of service.

This time I get an interesting issue:

I suppose tensorflow-gpu is an Eagle specific module that is loaded due to some input in the tutorial code? Does this error ring any bell for you @LourensVeen

If not, I'll proceed to develop a workaround for it :).

[djgroen]

Install output looks like this:

[djgroen]

Oh wait, MUSCLE is already there... although 0.1.0 is there?

Could it be the case that 0.2.0 doesn't work with Python 3.5? My install script looks like this:

[LourensVeen]

Ouch, sorry, I didn't see these as the mail got caught in a mail filter. The filter has been summarily deleted.

I'm not sure why tensorflow would be loaded, MUSCLE doesn't do any machine learning that I know of, and I don't see why any of its dependencies would :). Could it be that you have something else installed that uses it somehow?

It seems to me that there's a bit of a messy situation now in your Eagle account, with different versions of MUSCLE 3 installed in different places. And virtualenvs import the user and system environments unless you tell them not to, so that would confuse things further.

In order to clean up, could you do a pip3 uninstall muscle3 without any virtualenvs active? Then maybe do a module load python and uninstall again, just to be sure that that doesn't activate yet another environment? That should get rid of any existing old versions of MUSCLE.

Then a fresh install should be:

$ module load python
$ python3 -m venv /home/users/derek/Envs/muscle3    # or wherever you want to put it, but be sure to make a new one
$ . /home/users/derek/Envs/muscle3/bin/activate
$ pip3 install muscle3

Of course, that doesn't fix the from typing import Type issue if that still shows up. You already tried a newer version of typing, which didn't fix it. I don't understand why this does work on Python 3.4 in the YAtiML CI, but not on 3.5.1 on Eagle. I may be able to fix it by separating out the import of Type, then catching the exception and just defining it as Any.

[LourensVeen]

Looking at that again, there seems to be an issue with compatibility of numpy versions as well. In https://github.com/multiscale/muscle3/issues/32 we have a similar situation with version compatibility of grpcio. Things are confusing enough here already, so let's split that off as https://github.com/multiscale/muscle3/issues/34.

[LourensVeen]

I've been digging into the ImportError: cannot import name 'Type' issue. I can reproduce it on Python 3.5.1. It seems that the problem is that the typing module was introduced in Python 3.5.0, but this was an early version that is different in quite a few ways from the current version, including a lack of support for Type, TYPE_CHECKING, and NewType and a more restrictive `overload', plus at least one missing (compared to later versions) attribute for container types.

For Python 3.4.x, you can (and we do) install the typing module separately, which will add a recent version of typing so that everything works. For Python 3.5.0 and later however, the separate typing module does not do anything, you can install it, but you'll still get the packaged one. So this explains why that didn't work. Type is available from Python 3.5.2 on (and also 3.6.2, as 3.6.0 has a similar issue apparently). For those versions, you can also import Type from typing_extensions, which does solve that problem.

Unfortunately, YAtiML uses typing not only for type checking, but as a fundamental part of what it does, and it breaks internally because of missing features in those early typing versions. I don't see a good way of working around that.

It seems from https://github.com/python/mypy/issues/1838#issuecomment-231539460 that it may be possible to force loading the backport on <3.5.2 and <3.6.2 by manipulating the Python path before loading, although it would somehow have to figure out where the backport is installed. It seems like that may end up being a big kludge.

Another option may be to just declare early 3.5 and 3.6 versions unsupported, and ask anyone who has a system with such a version to ask their sysadmin to upgrade, or to use conda to get a newer one. 3.5 is already security-fix only and will be supported until the end of this year, with 3.6 obsolete by the end of 2021.

Are there any other Python versions available on Eagle?

[LourensVeen]

Good news! I've done some work on YAtiML, and I think I've managed to make it work on Python 3.5.1. At least it's passing the test suite now. I'll need to make a new release of that, and then a new release of yMMSL-Python using it, and then a new release of MUSCLE 3 using that. That'll probably be MUSCLE 3 0.3.1, which will add grid support, and then I can add this fix too.

[LourensVeen]

Fix released with 0.3.1, now we just need some more testing on Eagle to make sure everything works there too.

[LourensVeen]

MUSCLE3 has now been installed and run successfully on Eagle by at least one user, so I'm going to close this issue. Please reopen if you have any additional problems.