Closed tsafs closed 8 years ago
akorosov
commented
8 years ago Hi @Fast007 !
Could you try using conda-forge channel to install Nansat dependencies?
conda install -c conda-forge numpy scipy gdal basemap opencv netcdf4 proj4
You may need to set one more environment variable
export CPL_ZIP_ENCODING=UTF-8
We are experimenting with using this channel instead of maintaining our own. Your feedback is highly appreciated!
tsafs
commented
8 years ago Hi @akorosov Thanks for your quick reply!
Installing the dependencies via conda-forge almost went fine. nansat-gdal requires proj.4 instead of proj4. proj.4 is not available in the standard miniconda/conda-forge repositories, so I installed it from the anaconda repositories with
conda install -c anaconda proj.4which gives following output:
Package plan for installation in environment /home/user/miniconda:
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-env-2.5.1 | py27_0 26 KB anaconda
conda-4.1.4 | py27_0 197 KB anaconda
------------------------------------------------------------
Total: 224 KB
The following NEW packages will be INSTALLED:
proj.4: 4.9.1-0 anaconda
The following packages will be SUPERCEDED by a higher-priority channel:
conda: 4.1.4-py27_0 --> 4.1.4-py27_0 anaconda
conda-env: 2.5.1-py27_0 --> 2.5.1-py27_0 anacondaThen installing nansat-gdal with:
conda install -q --yes -c https://conda.anaconda.org/nersc nansat-gdalgives the following:
The following specifications were found to be in conflict:
- nansat-gdal
- opencv -> libpng 1.5.13
- opencv -> numpy 1.5*|1.6*|1.7*|1.8*
- opencv -> python 2.6*
Use "conda info <package>" to see the dependencies for each package.Note that I cannot install nansat-gdal without installing proj.4 instead of proj4.
Thanks for your help! I'm really thrown up here since I'm hitting the boundaries of my *nixness
akorosov
commented
8 years ago Hi @Fast007 ,
What I suggested was to use the conda-forge channel solely. It does have gdal, proj4, opencv, etc...
I.e. there is no need to install nansat-gdal
The easiest would be to make a fresh installation of conda (simply remove the old miniconda directory and launch miniconda.sh again). I believe (testing at the moment) that the following should work:
rm miniconda2 -rf
./miniconda.sh -b -f -p $HOME/miniconda2
export PYTHONPATH=
export PATH=$HOME/miniconda2/bin/:$PATH
conda update conda
conda install -c conda-forge basemap gdal opencv scipy nose
export GDAL_DATA=$HOME/miniconda2/share/gdal/
export GEOS_DIR=$HOME/miniconda2/
pip install https://github.com/nansencenter/nansat/archive/master.tar.gz
nosetests nansatHi @akorosov, Doing what you suggested gives me following error:
Collecting https://github.com/nansencenter/nansat/archive/master.tar.gz
Downloading https://github.com/nansencenter/nansat/archive/master.tar.gz (1.1MB)
100% |████████████████████████████████| 1.1MB 352kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-jjPKUX-build/setup.py", line 49, in <module>
raise ImportError(import_error_msg % 'gdal')
ImportError: Nansat requires gdal, which should be installed separately
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-jjPKUX-build/Here's also what conda list gives me:
# packages in environment at /home/user/miniconda:
#
basemap 1.0.7 np18py27_0
cairo 1.12.18 0
conda 4.1.4 py27_0
conda-env 2.5.1 py27_0
curl 7.30.0 0
dateutil 2.4.1 py27_0
freetype 2.4.10 0
gdal 2.0.0 py27_1
geos 3.3.3 0
hdf4 4.2.11 0
hdf5 1.8.9 1
jbig 2.1 0
jpeg 8d 0
kealib 1.4.5 0
libgdal 2.0.0 1
libgfortran 1.0 0
libnetcdf 4.2.1.1 1
libpng 1.5.13 1
libtiff 4.0.6 2
matplotlib 1.4.0 np18py27_0
netcdf4 1.1.1 np18py27_0
nose 1.3.7 py27_1
numpy 1.8.2 py27_1
opencv 2.4.6 np18py27_0
openssl 1.0.1k 1
pillow 2.7.0 py27_0
pip 8.1.1 py27_1
pixman 0.26.2 0
proj4 4.9.2 0
py2cairo 1.10.0 py27_2
pycosat 0.6.1 py27_0
pycrypto 2.6.1 py27_0
pyparsing 2.0.1 py27_0
pyqt 4.10.4 py27_0
python 2.7.10 1
pytz 2016.4 py27_0
pyyaml 3.11 py27_1
qt 4.8.5 0
readline 6.2 2
requests 2.9.1 py27_0
ruamel_yaml 0.11.7 py27_0
scipy 0.14.0 np18py27_0
setuptools 20.3 py27_0
sip 4.15.5 py27_0
six 1.10.0 py27_0
sqlite 3.9.2 0
tk 8.5.18 0
wheel 0.29.0 py27_0
xerces-c 3.1.2 0
xz 5.2.2 0
yaml 0.1.6 0
zlib 1.2.8 0 Edit:
If this helps, logging into the python console and importing gdal manually gives the following:
Python 2.7.10 |Continuum Analytics, Inc.| (default, Sep 15 2015, 14:27:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import gdal
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/miniconda/lib/python2.7/site-packages/gdal.py", line 2, in <module>
from osgeo.gdal import deprecation_warn
File "/home/user/miniconda/lib/python2.7/site-packages/osgeo/__init__.py", line 21, in <module>
_gdal = swig_import_helper()
File "/home/user/miniconda/lib/python2.7/site-packages/osgeo/__init__.py", line 17, in swig_import_helper
_mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: libgeos-3.4.2.so: cannot open shared object file: No such file or directoryEdit2: I think my issue is related to this.
tsafs
commented
8 years ago Hi @akorosov,
I was wondering if you could send me an overview of the installed packages on your system with conda list, if you used anaconda. Also, did you yourself install OpenCV with or without anaconda?
I seem to get nansat working following the steps in the official guide. Using anaconda, I cannot add OpenCV to the stack due to dependencies. After a manual installation of OpenCV nansat's tests break.
Thanks in advance.
akorosov
commented
8 years ago Hi @Fast007 ,
it seems that your are using a wrong environment of conda. If you followed my recipe exactly you should now have two folders in HOME: miniconda and miniconda2. You can switch between those by choosing the correct path to exectutable. Either you launch conda with:
$HOME/miniconda2/bin/conda
or you first set your PATH variable and then let bash automatically select the correct conda exectubale from the first available PATH:
export PATH=$HOME/miniconda2/bin/:$PATH
condaThe same with python, ipython, pip, gdalinfo and other executbales. In order to check, which executable is used you can use command which conda or which python.
Here is what I do:
~>echo $PATH
/home/antonk/miniconda2/bin:/home/antonk/bin:/home/antonk/miniconda/bin:/usr/local/sbin:etc...The miniconda2/bin path is the first and all executables will be searched there first:
~>which conda
/home/antonk/miniconda2/bin/conda
~>which python
/home/antonk/miniconda2/bin/python
~>which pip
/home/antonk/miniconda2/bin/pipconda update condaconda install -c conda-forge basemap gdal opencv scipy noseAnd this channel is also indicated in the list of available packages:
:~>conda list
# packages in environment at /home/antonk/miniconda2:
#
conda 4.1.4 py27_0
conda-env 2.5.1 py27_0
basemap 1.0.8.dev0 np111py27_1 conda-forge
blas 1.1 openblas conda-forge
curl 7.49.1 0 conda-forge
cycler 0.10.0 py27_0 conda-forge
expat 2.1.0 1 conda-forge
fontconfig 2.11.1 3 conda-forge
freetype 2.6.3 1 conda-forge
freexl 1.0.2 1 conda-forge
functools32 3.2.3.2 py27_1 conda-forge
gdal 2.1.0 np111py27_6 conda-forge
geos 3.4.2 2 conda-forge
giflib 5.1.2 1 conda-forge
hdf4 4.2.11 4 conda-forge
hdf5 1.8.17 2 conda-forge
icu 56.1 3 conda-forge
jasper 1.900.1 2 conda-forge
jpeg 9b 0 conda-forge
json-c 0.12 0 conda-forge
kealib 1.4.6 2 conda-forge
krb5 1.14.2 0 conda-forge
libiconv 1.14 2 conda-forge
libnetcdf 4.4.0 1 conda-forge
libpng 1.6.23 0 conda-forge
libspatialite 4.3.0a 8 conda-forge
libtiff 4.0.6 5 conda-forge
libxml2 2.9.3 8 conda-forge
matplotlib 1.5.1 np111py27_4 conda-forge
nose 1.3.7 py27_0 conda-forge
numpy 1.11.0 py27_blas_openblas_201 [blas_openblas] conda-forge
openblas 0.2.18 2 conda-forge
opencv 2.4.12 np111py27_2 conda-forge
openjpeg 2.1.0 5 conda-forge
proj.4 4.9.2 0 conda-forge
pyparsing 2.1.1 py27_0 conda-forge
pyproj 1.9.5.1 py27_0 conda-forge
pyshp 1.2.3 py27_0 conda-forge
python-dateutil 2.5.2 py27_0 conda-forge
pytz 2016.3 py27_0 conda-forge
scipy 0.17.1 np111py27_blas_openblas_200 [blas_openblas] conda-forge
sip 4.18 py27_0 conda-forge
six 1.10.0 py27_0 conda-forge
xz 5.0.5 1 conda-forge
libgcc 5.2.0 0
libgfortran 3.0.0 1
openssl 1.0.2h 1
pip 8.1.2 py27_0
postgresql 9.1.4 1
pycosat 0.6.1 py27_1
pycrypto 2.6.1 py27_0
pyqt 4.11.4 py27_3
python 2.7.11 5
pyyaml 3.11 py27_4
qt 4.8.7 3
readline 6.2 2
requests 2.10.0 py27_0
ruamel_yaml 0.11.7 py27_0
setuptools 23.0.0 py27_0
sqlite 3.13.0 0
tk 8.5.18 0
wheel 0.29.0 py27_0
xerces-c 3.1.2 0
yaml 0.1.6 0
zlib 1.2.8 3~>which pip
/home/antonk/miniconda2/bin/pip
~>pip install https://github.com/nansencenter/nansat/archive/master.tar.gz
Collecting https://github.com/nansencenter/nansat/archive/master.tar.gz
Downloading https://github.com/nansencenter/nansat/archive/master.tar.gz
/ 1.1MB 121.7MB/s
Collecting Pillow (from nansat==0.6.12)
Downloading Pillow-3.2.0.zip (10.5MB)
100% -------------- 10.5MB 162kB/s
Requirement already satisfied (use --upgrade to upgrade): pythesint in ./py/py-thesaurus-interface (from nansat==0.6.12)
Collecting cfunits (from nansat==0.6.12)
Requirement already satisfied (use --upgrade to upgrade): PyYAML in ./miniconda2/lib/python2.7/site-packages (from pythesint->nansat==0.6.12)
Building wheels for collected packages: Pillow
Running setup.py bdist_wheel for Pillow ... done
Stored in directory: /home/antonk/.cache/pip/wheels/88/2d/ce/3ff4ae4e2b8600d1bde1cbde5dfcc6d8770222c38348fe9139
Successfully built Pillow
Installing collected packages: Pillow, cfunits, nansat
Found existing installation: nansat 0.7.dev0
Can't uninstall 'nansat'. No files were found to uninstall.
Running setup.py install for nansat ... done
Successfully installed Pillow-3.2.0 cfunits-1.1.4 nansat-0.7.dev0conda install -c conda-forge netcdf4~>which nosetests
/home/antonk/miniconda2/bin/nosetests
~>nosetests nansat
....................../home/antonk/py/nansat/nansat/figure.py:302: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 200 but corresponding boolean dimension is 199
self.array[0][maskIndeces] = availIndeces[i]
../home/antonk/py/nansat/nansat/figure.py:302: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 600 but corresponding boolean dimension is 599
self.array[0][maskIndeces] = availIndeces[i]
.10:14:54|40|test_figure|test_get_auto_ticks_number|[ 27.14421637 28.07306492 29.00191348 29.93076204 30.85961059]
10:14:54|40|test_figure|test_get_auto_ticks_number|[ 70.36035747 70.65609906 70.95184065 71.24758224 71.54332383]
..10:14:55|40|test_figure|test_get_tick_index_from_grid|[19, 56]
10:14:55|40|test_figure|test_get_tick_index_from_grid|[132, 11]
....../home/antonk/py/nansat/nansat/mosaic.py:255: RuntimeWarning: invalid value encountered in sqrt
np.square(avg) * cntMat) / cntMat)
/home/antonk/miniconda2/lib/python2.7/site-packages/scipy/io/netcdf.py:297: RuntimeWarning: Cannot close a netcdf_file opened with mmap=True, when netcdf_variables or arrays referring to its data still exist. All data arrays obtained from such files refer directly to data on disk, and must be copied before the file can be cleanly closed. (See netcdf_file docstring for more information on mmap.)
), category=RuntimeWarning)
./home/antonk/py/nansat/nansat/mosaic.py:328: RuntimeWarning: invalid value encountered in less
bandArray[bandArray < vmin] = np.nan
/home/antonk/py/nansat/nansat/mosaic.py:329: RuntimeWarning: invalid value encountered in greater
bandArray[bandArray > vmax] = np.nan
/home/antonk/miniconda2/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:740: RuntimeWarning: All-NaN slice encountered
warnings.warn("All-NaN slice encountered", RuntimeWarning)
............./home/antonk/py/nansat/nansat/nansat.py:1911: UserWarning:
Python is started with -pylab option, transect will not work.
Please restart python without -pylab.
Please restart python without -pylab.''')
.....Warning 6: driver GTiff does not support creation option WRITE_BOTTOMUP
.Warning 6: driver GTiff does not support creation option WRITE_BOTTOMUP
.Warning 1: creating geographic file without lon/lat values!
.Warning 1: creating geographic file without lon/lat values!
.Warning 6: driver GTiff does not support creation option WRITE_BOTTOMUP
.Warning 1: creating geographic file without lon/lat values!
..Warning 1: creating geographic file without lon/lat values!
.Warning 1: creating geographic file without lon/lat values!
.Warning 1: creating geographic file without lon/lat values!
.............10:15:05|40|nansat|get_transect|Wrong band name 10
.............../home/antonk/py/nansat/nansat/figure.py:653: RuntimeWarning: invalid value encountered in divide
(self.cmax[iBand] - self.cmin[iBand]))
..............Warning 1: creating geographic file without lon/lat values!
..................................
----------------------------------------------------------------------
Ran 137 tests in 20.687s
OKHope that helps!
tsafs
commented
8 years ago Hi @akorosov, thanks for the detailed answer and guidance! I think the overall problem here is that I'm using a 32-bit machine, while you're using 64-bit. The versions of our installed packages differ (see below).
This is what I did:
$HOME/miniconda.sh -b -f -p $HOME/miniconda
export PATH=$HOME/miniconda/bin/:$PATH
conda update conda
conda install --yes -c conda-forge basemap gdal opencv scipy nose netcdf4
pip install https://github.com/nansencenter/nansat/archive/master.tar.gzThis is what I got:
Collecting https://github.com/nansencenter/nansat/archive/master.tar.gz
Downloading https://github.com/nansencenter/nansat/archive/master.tar.gz (1.1MB)
100% |████████████████████████████████| 1.1MB 263kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-aU_ayV-build/setup.py", line 49, in <module>
raise ImportError(import_error_msg % 'gdal')
ImportError: Nansat requires gdal, which should be installed separately
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-aU_ayV-build/Some more info:
user@ubuntu:~$ which conda
/home/user/miniconda/bin//conda
user@ubuntu:~$ which python
/home/user/miniconda/bin//python
user@ubuntu:~$ which pip
/home/user/miniconda/bin//pip
user@ubuntu:~$ conda list
# packages in environment at /home/user/miniconda:
#
basemap 1.0.7 np18py27_0
cairo 1.12.18 0
conda 4.1.5 py27_0
conda-env 2.5.1 py27_0
curl 7.30.0 0
dateutil 2.4.1 py27_0
freetype 2.4.10 0
gdal 2.0.0 py27_1
geos 3.3.3 0
hdf4 4.2.11 0
hdf5 1.8.9 1
jpeg 8d 0
kealib 1.4.5 0
libgdal 2.0.0 1
libgfortran 1.0 0
libnetcdf 4.2.1.1 1
libpng 1.5.13 1
matplotlib 1.4.0 np18py27_0
netcdf4 1.1.1 np18py27_0
nose 1.3.7 py27_1
numpy 1.8.2 py27_1
opencv 2.4.6 np18py27_0
openssl 1.0.1k 1
pip 8.1.1 py27_1
pip 8.1.2 <pip>
pixman 0.26.2 0
proj4 4.9.2 0
py2cairo 1.10.0 py27_2
pycosat 0.6.1 py27_0
pycrypto 2.6.1 py27_0
pyparsing 2.0.1 py27_0
pyqt 4.10.4 py27_0
python 2.7.10 1
pytz 2016.4 py27_0
pyyaml 3.11 py27_1
qt 4.8.5 0
readline 6.2 2
requests 2.9.1 py27_0
ruamel_yaml 0.11.7 py27_0
scipy 0.14.0 np18py27_0
setuptools 20.3 py27_0
sip 4.15.5 py27_0
six 1.10.0 py27_0
sqlite 3.13.0 0
tk 8.5.18 0
wheel 0.29.0 py27_0
xerces-c 3.1.2 0
yaml 0.1.6 0
zlib 1.2.8 0 Yes, conda-forge provides only 64-bit versions. We do have nansat-gdal for 32-bit, but how to install opencv the easiest way?.. Let me check.
akorosov
commented
8 years ago @Fast007 , what is the main reason to use 32-bits machine? An old PC? SeaIceDrift is quite demanding for resources since one image is 10000x10000 pixels. Will the PC manage processing?
tsafs
commented
8 years ago Alright, I was able to get a 64-bit machine! Everything is solved now. Thank you very much!
akorosov
commented
8 years ago Nice to hear! ;) And I realized that it is quite complicated to solve the incompatibility issue of nansat-gdal and opencv. That is one of the reasons to switch to conda-forge.
tsafs
commented
8 years ago In case anyone else has issues with the installation I will post the exact steps I took to get this running. Note: I'm doing this on a 64-bit Red Hat Linux machine.
# download the latest version of miniconda
wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
# make it executable
chmod +x miniconda.sh
# install it
$HOME/miniconda.sh -b -f -p $HOME/miniconda
# change the current python/pip/conda paths to the following.
export PATH=$HOME/miniconda/bin/:$PATH
# update conda
conda update conda
# install all dependencies
conda install --yes -c conda-forge basemap gdal opencv scipy nose netcdf4
# install nansat
pip install https://github.com/nansencenter/nansat/archive/master.tar.gz
# configure environment
export GDAL_DATA=$HOME/miniconda/share/gdal/
export GEOS_DIR=$HOME/miniconda/
# run tests
nosetests nansatI had problems running the export command. My system was using the tcsh shell, which I found out by running echo $SHELL. export was not accessible there. To access export I simply logged into the bash shell by typing /bin/bash. Then I ran the mentioned export-commands above.
I'm having trouble installing the required prerequisites.
First I tried installing them with a windows machine, which was not possible, because there were dependencies of nansat, that were not available for windows.
Then I setup a VM running Ubuntu 16.04. I installed nansat just the way they explained in their on-site guidance: using miniconda. Everything worked fine this far - everything was setup, except for OpenCV. There begins the trouble. Firstly, since I was using miniconda, I thought why not install OpenCV through conda? Well, that wasn't possible. I got a dependency issue which follows:
Running (this gives opencv version 2.4.6)
gives
and running (this gives opencv version 2.4.9.1)
gives
It looks like OpenCV works with other dependencies than nansat.
Which OpenCV version did you use? How did you install it? Did you set it up from source?