search

gafusion / omas

Ordered Multidimensional Array Structure
http://gafusion.github.io/omas
MIT License
30 stars 15 forks source link

pip install of omas on iris #111

Closed jmcclena closed 4 years ago

jmcclena commented 4 years ago

pip install of omas on iris is failing for me with the following error:

pip install --upgrade -e .[build_structures,build_documentation]
....
....
 Using cached https://files.pythonhosted.org/packages/b2/4c/b6f966ac91c5670ba4ef0b0b5613b5379e3c7abdfad4e7b89a87d73bae13/pandas-0.24.2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-Y9FfVI/pandas/setup.py", line 746, in <module>
        **setuptools_kwargs)
      File "/fusion/usc/opt/python/2.7.11/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 269, in __init__
      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 313, in fetch_build_eggs
      File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 825, in resolve

      File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1070, in best_match

      File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1082, in obtain

      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 380, in fetch_build_egg
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 640, in easy_install
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 670, in install_item
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 853, in install_eggs
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1081, in build_and_install
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1067, in run_setup
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 246, in run_setup
      File "/fusion/usc/opt/python/2.7.11/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 195, in setup_context
      File "/fusion/usc/opt/python/2.7.11/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 166, in save_modules
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 141, in resume
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 154, in save_modules
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 195, in setup_context
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 243, in run_setup
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 273, in run
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 242, in runner
      File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 45, in _execfile
      File "/tmp/easy_install-duGNUu/numpy-1.19.0rc1/setup.py", line 182
        raise ValueError(f'Submodule {p} missing')
                                                ^
    SyntaxError: invalid syntax

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Y9FfVI/pandas/
You are using pip version 18.1, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
orso82 commented 4 years ago

looks like you are using python2? In the error I read:

/fusion/usc/opt/python/2.7.11/lib/python2.7/distutils/core.py

Can you try:

module purge
module load omfit/unstable
cd folder_where_you_have_installed_omas
pip install --user --upgrade -e .[build_structures,build_documentation]
jmcclena commented 4 years ago

Then I get the error:

[mcclenaghanj@irisa omas]$ module purge
[mcclenaghanj@irisa omas]$ module load omfit/unstable
[mcclenaghanj@irisa omas]$ pip install --upgrade -e .[build_structures,build_documentation]
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/mcclenaghanj/omas
Requirement already satisfied, skipping upgrade: numpy>=1.16.1 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (1.18.4)
Requirement already satisfied, skipping upgrade: uncertainties in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (3.1.2)
Requirement already satisfied, skipping upgrade: pint in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (0.11)
Requirement already satisfied, skipping upgrade: netCDF4 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (1.5.3)
Requirement already satisfied, skipping upgrade: boto3 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (1.13.11)
Requirement already satisfied, skipping upgrade: matplotlib in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (3.1.1)
Requirement already satisfied, skipping upgrade: scipy in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (1.4.1)
Requirement already satisfied, skipping upgrade: h5py in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (2.10.0)
Requirement already satisfied, skipping upgrade: pymongo in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (3.10.1)
Requirement already satisfied, skipping upgrade: dnspython in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (1.16.0)
Requirement already satisfied, skipping upgrade: xmltodict in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (0.12.0)
Requirement already satisfied, skipping upgrade: xarray in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (0.12.0)
Requirement already satisfied, skipping upgrade: setuptools>=41.2 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (46.4.0.post20200518)
Requirement already satisfied, skipping upgrade: Sphinx in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (3.0.3)
Requirement already satisfied, skipping upgrade: sphinx-bootstrap-theme in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (0.8.0)
Requirement already satisfied, skipping upgrade: sphinx-gallery in /home/mcclenaghanj/.local/lib/python3.7/site-packages (from omas==0.57.0) (0.7.0)
Requirement already satisfied, skipping upgrade: Pillow in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from omas==0.57.0) (6.2.1)
Requirement already satisfied, skipping upgrade: bs4 in /home/mcclenaghanj/.local/lib/python3.7/site-packages (from omas==0.57.0) (0.0.1)
Requirement already satisfied, skipping upgrade: cftime in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from netCDF4->omas==0.57.0) (1.1.3)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from boto3->omas==0.57.0) (0.10.0)
Requirement already satisfied, skipping upgrade: botocore<1.17.0,>=1.16.11 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from boto3->omas==0.57.0) (1.16.11)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from boto3->omas==0.57.0) (0.3.3)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from matplotlib->omas==0.57.0) (1.2.0)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from matplotlib->omas==0.57.0) (2.8.1)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from matplotlib->omas==0.57.0) (0.10.0)
Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from matplotlib->omas==0.57.0) (2.4.7)
Requirement already satisfied, skipping upgrade: six in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from h5py->omas==0.57.0) (1.14.0)
Requirement already satisfied, skipping upgrade: pandas>=0.19.2 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from xarray->omas==0.57.0) (1.0.3)
Requirement already satisfied, skipping upgrade: sphinxcontrib-qthelp in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.0.3)
Requirement already satisfied, skipping upgrade: Pygments>=2.0 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (2.6.1)
Requirement already satisfied, skipping upgrade: babel>=1.3 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (2.8.0)
Requirement already satisfied, skipping upgrade: Jinja2>=2.3 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (2.11.2)
Requirement already satisfied, skipping upgrade: sphinxcontrib-htmlhelp in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.0.3)
Requirement already satisfied, skipping upgrade: alabaster<0.8,>=0.7 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (0.7.12)
Requirement already satisfied, skipping upgrade: imagesize in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.2.0)
Requirement already satisfied, skipping upgrade: snowballstemmer>=1.1 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (2.0.0)
Requirement already satisfied, skipping upgrade: sphinxcontrib-serializinghtml in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.1.4)
Requirement already satisfied, skipping upgrade: sphinxcontrib-devhelp in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.0.2)
Requirement already satisfied, skipping upgrade: requests>=2.5.0 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (2.23.0)
Requirement already satisfied, skipping upgrade: sphinxcontrib-applehelp in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.0.2)
Requirement already satisfied, skipping upgrade: sphinxcontrib-jsmath in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (1.0.1)
Requirement already satisfied, skipping upgrade: docutils>=0.12 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (0.15.2)
Requirement already satisfied, skipping upgrade: packaging in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Sphinx->omas==0.57.0) (20.1)
Requirement already satisfied, skipping upgrade: beautifulsoup4 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from bs4->omas==0.57.0) (4.9.1)
Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from botocore<1.17.0,>=1.16.11->boto3->omas==0.57.0) (1.25.9)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from pandas>=0.19.2->xarray->omas==0.57.0) (2019.3)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from Jinja2>=2.3->Sphinx->omas==0.57.0) (1.1.1)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from requests>=2.5.0->Sphinx->omas==0.57.0) (2.9)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from requests>=2.5.0->Sphinx->omas==0.57.0) (3.0.4)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from requests>=2.5.0->Sphinx->omas==0.57.0) (2020.4.5.1)
Requirement already satisfied, skipping upgrade: soupsieve>1.2 in /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/miniconda3_20200421/lib/python3.7/site-packages (from beautifulsoup4->bs4->omas==0.57.0) (2.0.1)
Installing collected packages: omas
  Attempting uninstall: omas
    Found existing installation: omas 0.57.0
    Uninstalling omas-0.57.0:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'INSTALLER'
Check the permissions.
orso82 commented 4 years ago

@jmcclena note that my command had --user in it ;) That should install OMAS in your ~/.local directory, where you have write permissions

orso82 commented 4 years ago

If that also fails, try adding --no-deps to the mix.

jmcclena commented 4 years ago 

[mcclenaghanj@irisa omas]$ pip install --user --upgrade --no-deps  -e .[build_structures,build_documentation]
Obtaining file:///home/mcclenaghanj/omas
Installing collected packages: omas
  Attempting uninstall: omas
    Found existing installation: omas 0.57.0
    Uninstalling omas-0.57.0:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'INSTALLER'
Check the permissions.
``
orso82 commented 4 years ago

ok, how about: pip install --user --no-deps --ignore-installed --upgrade -e .[build_structures,build_documentation]

orso82 commented 4 years ago

@jmcclena I assume we figured this one out. If not, reopen, and we can iterate more.

jmcclena commented 4 years ago

Nope :(

[mcclenaghanj@irisc omas]$ pip install --user --no-deps --ignore-installed --upgrade -e .
Obtaining file:///home/mcclenaghanj/omas
Installing collected packages: omas
  Attempting uninstall: omas
    Found existing installation: omas 0.57.0
    Uninstalling omas-0.57.0:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'INSTALLER'
Check the permissions.

If I remove export PATH="/home/mcclenaghanj/omas:$PATH" from by bash_profile (and readd it after installation) and load python3 instead of omfit/unstable, it looks like in installs. However, I can't seem to get omfit to pick up my version of omas.



[mcclenaghanj@irisc omas]$ pip install --user --no-deps --ignore-installed --upgrade -e .
Obtaining file:///home/mcclenaghanj/omas
Installing collected packages: omas
  Found existing installation: omas 0.57.0
    Uninstalling omas-0.57.0:
      Successfully uninstalled omas-0.57.0
  Running setup.py develop for omas
Successfully installed omas
``
orso82 commented 4 years ago

@orso82 you mean export PYTHONPATH="/home/mcclenaghanj/omas:$PYTHONPATH" right? Well, that should not be necessary. Doing a pip install --user should install OMAS in your '~/.local/` folder.

When OMFIT starts it says what custom PYTHONPATH folders it is picking up.

================================================================================
Warning: The following user-defined paths are in your Python environment:
...
To use a clean Python environment set OMFIT_CLEAN_PYTHON_ENVIRONMENT=1
To suppress this warning message  set OMFIT_CLEAN_PYTHON_ENVIRONMENT=0
================================================================================

can you check what it says?

jmcclena commented 4 years ago

it was PATH, but PATH or PYTHON doesn't change anything.

Loading base utility functions...
================================================================================
Warning: The following user-defined paths have been removed from your Python environment:
  /fusion/projects/codes/atom/omfit_py3/atom_SATURN_GCC/OMFIT-source
  /fusion/projects/codes/atom/omfit_v1/atom_SATURN_GCC/gacode/cgyro/bin
  /fusion/projects/codes/atom/omfit_v1/atom_SATURN_GCC/gacode/gapy/f2py
  /fusion/projects/codes/atom/omfit_v1/atom_SATURN_GCC/gacode/python
  /home/mcclenaghanj/.local/lib/python3.7/site-packages
To use your original Python environment set OMFIT_CLEAN_PYTHON_ENVIRONMENT=0
================================================================================
External imports...
Sanity check Python packages r
orso82 commented 4 years ago

That I know of, I do not setting PATH makes sense... Python should only care about PYTHONPATH. It is interesting it makes a difference for PIP though. Maybe @smithsp undestands why?

anywas, I see that OMFIT detected /home/mcclenaghanj/.local/lib/python3.7/site-packages. Cna you make sure that omas is listed there? /home/mcclenaghanj/.local/lib/python3.7/site-packages/omas.

smithsp commented 4 years ago

I don't know what the problem is.