search

znicholls / CMIP6-json-data-citation-generator

Simple scripts to automatically generate json data citations for CMIP6 data files
https://cmip6-json-data-citation-generator.readthedocs.io/en/latest/
BSD 2-Clause "Simplified" License
1 stars 2 forks source link

Installation error #14

Open MartinaSt opened 6 years ago

MartinaSt commented 6 years ago

Hi @znicholls At last I have checked out the code. As I want to use the existing python2.7 on our HPC, I had to install missing packages in a local path:

python setup.py install --prefix=/home/dkrz/k204082/.local/

I got the following error:

...
Searching for pytest
Best match: pytest cov-2.5.1
Downloading https://files.pythonhosted.org/packages/24/b4/7290d65b2f3633db51393bdf8ae66309b37620bc3ec116c5e357e3e37238/pytest-cov-2.5.1.tar.gz#sha256=03aa752cf11db41d281ea1d807d954c4eda35cfa1b21d6971966cc041bbf6e2d
Processing pytest-cov-2.5.1.tar.gz
Writing /tmp/easy_install-ZAjOEj/pytest-cov-2.5.1/setup.cfg
Running pytest-cov-2.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ZAjOEj/pytest-cov-2.5.1/egg-dist-tmp-l4JpD2
warning: no files found matching '.isort.cfg'
warning: no files found matching '.pylintrc'
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
removing '/mnt/lustre01/pf/k/k204082/.local/lib/python2.7/site-packages/pytest_cov-2.5.1-py2.7.egg' (and everything under it)
creating /mnt/lustre01/pf/k/k204082/.local/lib/python2.7/site-packages/pytest_cov-2.5.1-py2.7.egg
Extracting pytest_cov-2.5.1-py2.7.egg to /mnt/lustre01/pf/k/k204082/.local/lib/python2.7/site-packages
pytest-cov 2.5.1 is already the active version in easy-install.pth

Installed /mnt/lustre01/pf/k/k204082/.local/lib/python2.7/site-packages/pytest_cov-2.5.1-py2.7.egg
error: Could not find required distribution pytest
znicholls commented 6 years ago

does 

pip install virtualenv
python -m virtualenv venv
source venv/bin/activate
python setup.py install

make your life any better? Doing it locally rather than with a venv seems full of danger...

znicholls commented 6 years ago

(As an fyi, I doubt you'll find anything interesting until we sort #13)

MartinaSt commented 6 years ago

I do not see an option for pip install, which allows me to install venv in a local directory (other than easy-install)...

znicholls commented 6 years ago

You may not need that step as virtualenv may already be installed? If venv isn't installed on your machine and you can't install it then this is going to make things trickier to debug

MartinaSt commented 6 years ago

Sorry: 

k204082@mlogin107 python -m virtualenv venv
/sw/rhel6-x64/python/python-2.7-ve0-gcc49/bin/python: No module named virtualenv
znicholls commented 6 years ago

Ok. Can you put the output of

''' python --version '''

And

''' pip list '''

Please

MartinaSt commented 6 years ago

Sure. The CMIP6-json-data-citation-generator is surprising as I am not allowed to write packages:

k204082@mlogin107 python -V
Python 2.7.9
k204082@mlogin107 pip list
alabaster (0.7.9)
astroid (1.4.7)
autobahn (0.10.8)
Babel (2.3.4)
backports.functools-lru-cache (1.2.1)
backports.ssl-match-hostname (3.4.0.2)
basemap (1.0.7)
bokeh (0.12.4)
buildbot-slave (0.8.12)
Cartopy (0.14.2)
cdat-lite (6.0rc2)
cdo (1.2.5)
certifi (2015.4.28)
cffi (0.9.2)
characteristic (14.3.0)
click (6.6)
cloudpickle (0.1.1)
CMIP6-json-data-citation-generator (1.0.0)
CMOR (2.0)
codecov (2.0.15)
configobj (5.0.6)
configparser (3.5.0)
cryptography (0.8.2)
cx-Oracle (5.2)
Cython (0.23)
dask (0.8.1)
decorator (4.0.2)
docutils (0.12)
dreqPy (1b23)
ecmwf-api-client (1.3)
enum34 (1.0.4)
esgf-pyclient (0.1.8)
ESMPy (700)
foolscap (0.8.0)
fusepy (2.0.4)
future (0.15.2)
futures (3.1.1)
GDAL (1.11.2)
Genshi (0.7)
gitdb2 (2.0.0)
GitPython (2.1.1)
gsw (3.0.3)
httplib2 (0.9.1)
humanize (0.5.1)
imagesize (0.7.1)
ipython (3.1.0)
isort (4.2.5)
Jinja2 (2.7.3)
jsonschema (2.4.0)
lazy-object-proxy (1.2.2)
lxml (3.4.4)
Mako (1.0.2)
MarkupSafe (0.23)
matplotlib (1.4.3)
mccabe (0.5.0)
mock (1.0.1)
mpi4py (2.0.0)
netCDF4 (1.1.8)
networkx (1.10)
nose (1.3.4)
numexpr (2.4.3)
numpy (1.9.2)
pandas (0.16.2)
Paste (2.0.2)
PasteDeploy (1.5.2)
PasteScript (2.0.2)
patsy (0.4.1)
pbr (1.8.0)
pexpect (3.3)
Pillow (2.9.0)
pip (6.0.8)
psycopg2 (2.6.2)
pupynere (1.0.15)
pyasn1 (0.1.7)
pyasn1-modules (0.0.5)
pycairo (1.8.8)
pycparser (2.12)
Pydap (3.1.1)
pyflakes (1.0.0)
Pygments (2.0.2)
pygraphviz (1.2)
pylint (1.6.4)
PyNGL (1.5.0b0)
PyNIO (1.5.0b0)
pyOpenSSL (0.15.1)
pyparsing (2.0.3)
pyproj (1.9.5.1)
pyresample (1.1.0)
Pyro (3.16)
pyshp (1.2.3)
pytest-cov (2.5.1)
python-dateutil (2.4.2)
python-swiftclient (3.3.1.dev26)
pytz (2015.2)
PyYAML (3.12)
pyzmq (14.6.0)
requests (2.13.0)
scikit-image (0.11.3)
scikits.statsmodels (0.3.1)
scipy (0.15.1)
seaborn (0.6.0)
seawater (3.3.3)
serpent (1.10)
service-identity (14.0.0)
setuptools (12.0.5)
Shapely (1.5.15)
simplejson (3.7.3)
six (1.9.0)
smmap2 (2.0.1)
snowballstemmer (1.2.1)
Sphinx (1.5a2)
SQLAlchemy (1.0.8)
sqlalchemy-migrate (0.10.0)
sqlparse (0.1.16)
statsmodels (0.8.0rc1)
swift-fuse (0.1)
tables (3.2.1)
tabulate (0.7.5)
Tempita (0.5.3.dev0)
toolz (0.7.4)
tornado (4.1)
Twisted (15.1.0)
txaio (1.1.0)
wrapt (1.10.8)
wxPython (3.0.2.0)
wxPython-common (3.0.2.0)
xarray (0.7.0)
XlsxWriter (0.8.4)
zope.interface (4.1.2)
znicholls commented 6 years ago

Hmm that is strange. So if you do, pip install virtualenv, nothing happens?

Can you also post the output of which python and which pip please?

MartinaSt commented 6 years ago

I tried the installation with the Makefile first, maybe that left something over...

My PYTHONPATH shows the system path to the python installation:

k204082@mlogin107 echo $PYTHONPATH /home/dkrz/k204082/mars/lib:/home/dkrz/k204082/.local/lib/python2.7/site-packages:/pf/k/k204082/src/DKRZ_CDP/submission_forms/dkrz_forms:/sw/rhel6-x64/python/python-2.7-ve0-gcc49/lib/python2.7

k204082@mlogin107 which python
/sw/rhel6-x64/python/python-2.7-ve0-gcc49/bin/python
k204082@mlogin107 which pip
/sw/rhel6-x64/python/python-2.7-ve0-gcc49/bin/pip
k204082@mlogin107 pip install virtualenv
...
  IOError: [Errno 13] Permission denied: '/sw/rhel6-x64/python/python-2.7-ve0-gcc49/lib/python2.7/site-packages/virtualenv.pyc'
znicholls commented 6 years ago

hmmm... trying to do this without virtualenv and a lack of permissions will be tricky. Does anyone at DKRZ use virtualenv or other environment managers?

znicholls commented 5 years ago

@MartinaSt is this fixed now?

AndreaLamm commented 5 years ago

Hi @znicholls, my name is Andrea and I'm working with Martina on the CMIP6 Citation service. I have tested your package including the virtual enviroment on our HPC. Just one adaptation was necessary; I had a problem with the routine "ModuleNotFoundError". Replaced by "ImportError" in the packages netcdf_scm: iris_cube_wrappers.py and utils.py , the testscript worked well.

znicholls commented 5 years ago

Hey Andrea,

Thanks for the feedback! Which version of Python and what operating system were you using? I'll see if I can reproduce and fix that bug in NetCDF-SCM :)