Trouble setting up python tools

[abaillod]

Hello,

It seems that the Slack channel is dead, so I am asking my question here.

I am painfully trying to set up the python tools on my machine - and, as a full python beginner, I naturally run into problems. I am using the master branch. I installed a virtual environment where I installed all necessary packages and set up the python path to SPEC/Utilities/pythontools/py_spec

abaillod@spcpc602:~/.virtualenvs/simsopt/bin> workon simsopt
(simsopt) abaillod@spcpc602:~/.virtualenvs/simsopt/bin> python
Python 3.6.9 (default, Oct 29 2019, 10:39:36) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/abaillod/SPEC/Utilities/pythontools/py_spec', '/home/abaillod/.virtualenvs/simsopt/lib/python36.zip', '/home/abaillod/.virtualenvs/simsopt/lib64/python3.6', '/home/abaillod/.virtualenvs/simsopt/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6', '/usr/lib/python3.6', '/home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages', '/home/abaillod/simsopt/src']

However, when trying to import py_spec, I get

>>> import py_spec
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages/py_spec/__init__.py", line 4, in <module>
    from .ci import test
  File "/home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages/py_spec/ci/__init__.py", line 3, in <module>
    import test
ModuleNotFoundError: No module named 'test'

Similarly, trying to run setup.py in /SPEC/Utilities/pythontools lead to

(simsopt) abaillod@spcpc602:~/SPEC/Utilities/pythontools> python setup.py
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from py_spec import __version__
  File "/home/abaillod/SPEC/Utilities/pythontools/py_spec/__init__.py", line 4, in <module>
    from .ci import test
  File "/home/abaillod/SPEC/Utilities/pythontools/py_spec/ci/__init__.py", line 3, in <module>
    import test
ModuleNotFoundError: No module named 'test'

It seems to me that python does not understand the relative path to .ci... What am I doing wrong? It seems to me that I messed up my set-up but I can't figure out how to make py_spec work properly. Is py_spec up-to-date on the master branch? Sorry if this is a very basic question!

[zhucaoxiang]

@abaillod The python relative import is actually tricky. The quickest fix is to install it to your system and remove the py_spec path from sys.path. You can install it via

pip install py_spec

Or go to the ./pythontools/ and do pip install -e ..

The first step install everything over the source in PyPI and the second one installs it from your local sources.

Of course, we should figure out how to resolve the relative import error here.

[abaillod]

Thanks Caoxiang for your help. I was able to install py_spec.

However the error still appears when trying to import the module:

(simsopt) abaillod@spcpc602:~> pip install py_spec
Collecting py_spec
  Using cached py_spec-3.0.1-py3-none-any.whl (21 kB)
Installing collected packages: py-spec
Successfully installed py-spec-3.0.1
(simsopt) abaillod@spcpc602:~> python
Python 3.6.9 (default, Oct 29 2019, 10:39:36) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import py_spec
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages/py_spec/__init__.py", line 4, in <module>
    from .ci import test
  File "/home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages/py_spec/ci/__init__.py", line 3, in <module>
    import test
ModuleNotFoundError: No module named 'test'

[zhucaoxiang]

What about the second option? I tried it and it works for me... Or try to use pip install -U py_spec in case there are local caches.

[abaillod]

Same, both with and without the option -U. I did pip uninstall py_spec first. I will try again on a clean new virtual environment...

(simsopt) abaillod@spcpc602:~/SPEC/Utilities/pythontools> pip install -U -e .
Obtaining file:///home/abaillod/SPEC/Utilities/pythontools
    ERROR: Command errored out with exit status 1:
     command: /home/abaillod/.virtualenvs/simsopt/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/abaillod/SPEC/Utilities/pythontools/setup.py'"'"'; __file__='"'"'/home/abaillod/SPEC/Utilities/pythontools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gdwjy4pc
         cwd: /home/abaillod/SPEC/Utilities/pythontools/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/abaillod/SPEC/Utilities/pythontools/setup.py", line 2, in <module>
        from py_spec import __version__
      File "/home/abaillod/SPEC/Utilities/pythontools/py_spec/__init__.py", line 4, in <module>
        from .ci import test
      File "/home/abaillod/SPEC/Utilities/pythontools/py_spec/ci/__init__.py", line 3, in <module>
        import test
    ModuleNotFoundError: No module named 'test'
    ----------------------------------------
WARNING: Discarding file:///home/abaillod/SPEC/Utilities/pythontools. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[abaillod]

I tested the installation with another PhD student here at SPC - he had the same problem on his own personnal laptop.

The work-around we came up with is to comment the line

from .ci import test

in ./Utilities/pythontools/py_spec. This is a quick and durty fix, I guess the real problem should be identified and fixed...

[smiet]

Thanks for bringing this up! This is an isue that I don't immediately see how to fix: we need to import the test script for the github actions, where we test the output by the following line:

          python3 -m py_spec.ci.test compare.h5 G2V32L1Fi.001.sp.h5 

(the -m option opens python and executes the test module as a script)

The trouble is that the test script is already a script, and the import fails if py_spec is installed via pypi).

I have to rewrite the ci/test script as a module, and then hopefully the import will not fail. I will try to do that later today.

Thank you for your patience, for now commenting the line should work for you.

[zhisong]

Will if __name__ =='__main__' help?

[smiet]

I have included the if __name__ =='__main__', and changed the import. Need to test if the new package on pypi is able to be imported now. (should be uploaded now as well, thanks @zhucaoxiang for automating this!)

@abaillod can you test if the pypi package works on your system by pip install --upgrade'ing and doing what you did before?

pull request is here; #125

the version of this package is 3.0.1b since pypi would not accept the changes under the same version number.

[abaillod]

It seems to work on my side. Thank's for the fix! I close this issue.