[BUG]: environment-gpu.yml can't create environments with newest conda and pip install doesn't work too!

dudeperf3ct commented 4 years ago

Installing solaris using conda

conda update conda
conda env create -f environment-gpu.yml

results in

visionosl:solaris$ conda env create -f environment-gpu.yml
Collecting package metadata (repodata.json): done
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                              \  

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda_env/cli/main_create.py", line 111, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda_env/installers/conda.py", line 32, in install
        prune=getattr(args, 'prune', False), update_modifier=UpdateModifier.FREEZE_INSTALLED)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 117, in solve_for_transaction
        should_retry_solve)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 158, in solve_for_diff
        force_remove, should_retry_solve)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 281, in solve_final_state
        ssc = self._run_sat(ssc)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 808, in _run_sat
        should_retry_solve=ssc.should_retry_solve
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 1318, in solve
        self.find_conflicts(specs, specs_to_add, history_specs)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 347, in find_conflicts
        bad_deps = self.build_conflict_map(specs, specs_to_add, history_specs)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 507, in build_conflict_map
        root, search_node, dep_graph, num_occurances)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 369, in breadth_first_search_for_dep_graph
        last_spec = MatchSpec.union((path[-1], target_paths[-1][-1]))[0]
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 481, in union
        return cls.merge(match_specs, union=True)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 475, in merge
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 475, in <lambda>
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 502, in _merge
        final = this_component.union(that_component)
      File "/home/visionosl/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 764, in union
        return '|'.join(options)
    TypeError: sequence item 0: expected str instance, Channel found

`$ /home/visionosl/anaconda3/bin/conda-env create -f environment-gpu.yml`

  environment variables:
                 CIO_TEST=<not set>
  CONDA_AUTO_UPDATE_CONDA=false
                CONDA_EXE=/home/visionosl/anaconda3/bin/conda
         CONDA_PYTHON_EXE=/home/visionosl/anaconda3/bin/python
               CONDA_ROOT=/home/visionosl/anaconda3
              CONDA_SHLVL=0
            DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
           MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
                     PATH=/home/visionosl/anaconda3/bin:/home/visionosl/anaconda3/condabin:/home
                          /visionosl/bin:/home/visionosl/.local/bin:/usr/local/sbin:/usr/local/b
                          in:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
            XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
         XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0

     active environment : None
            shell level : 0
       user config file : /home/visionosl/.condarc
 populated config files : /home/visionosl/.condarc
          conda version : 4.8.2
    conda-build version : 3.18.9
         python version : 3.7.4.final.0
       virtual packages : __cuda=10.1
                          __glibc=2.23
       base environment : /home/visionosl/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/visionosl/anaconda3/pkgs
                          /home/visionosl/.conda/pkgs
       envs directories : /home/visionosl/anaconda3/envs
                          /home/visionosl/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.4 Linux/4.15.0-70-generic ubuntu/16.04.6 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

Environment Information

Running this on a fresh installation of anaconda

$ conda list anaconda$
# packages in environment at /home/visionosl/anaconda3:
#
# Name                    Version                   Build  Channel
anaconda                  2019.10                  py37_0

$ conda info

     active environment : None
            shell level : 0
       user config file : /home/visionosl/.condarc
 populated config files : /home/visionosl/.condarc
          conda version : 4.8.2
    conda-build version : 3.18.9
         python version : 3.7.4.final.0
       virtual packages : __cuda=10.1
                          __glibc=2.23
       base environment : /home/visionosl/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/visionosl/anaconda3/pkgs
                          /home/visionosl/.conda/pkgs
       envs directories : /home/visionosl/anaconda3/envs
                          /home/visionosl/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.4 Linux/4.15.0-70-generic ubuntu/16.04.6 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Installing solaris using pip

$ sudo add-apt-repository ppa:ubuntugis/ppa
$ sudo apt-get update
$ sudo apt-get install python-numpy gdal-bin libgdal-dev python3-rtree
$ pip install solaris

results in gdal error

Failed to build gdal
ERROR: torchvision 0.5.0 has requirement torch==1.4.0, but you'll have torch 1.3.1 which is incompatible.
Installing collected packages: pyparsing, numpy, kiwisolver, cycler, matplotlib, shapely, gdal, torch, pillow, torchvision, urllib3, pyproj, decorator, networkx, pyyaml, imageio, scipy, PyWavelets, scikit-image, opencv-python, affine, click-plugins, attrs, munch, cligj, fiona, termcolor, absl-py, markdown, grpcio, werkzeug, tensorboard, gast, mock, tensorflow-estimator, h5py, keras-applications, astor, keras-preprocessing, tensorflow, rtree, snuggs, rasterio, pytz, pandas, geopandas, mercantile, supermercado, rio-cogeo, imgaug, opencv-python-headless, albumentations, solaris
    Running setup.py install for gdal ... error
    ERROR: Command errored out with exit status 1:
     command: /home/visionosl/anaconda3/envs/dl/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-hosc2bjy/gdal/setup.py'"'"'; __file__='"'"'/tmp/pip-install-hosc2bjy/gdal/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-otn602ug/install-record.txt --single-version-externally-managed --compile --install-headers /home/visionosl/anaconda3/envs/dl/include/python3.7m/gdal
         cwd: /tmp/pip-install-hosc2bjy/gdal/
    Complete output (62 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    copying gdal.py -> build/lib.linux-x86_64-3.7
    copying ogr.py -> build/lib.linux-x86_64-3.7
    copying osr.py -> build/lib.linux-x86_64-3.7
    copying gdalconst.py -> build/lib.linux-x86_64-3.7
    copying gdalnumeric.py -> build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/gdal.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/gdalconst.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/osr.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/gdal_array.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/gnm.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/__init__.py -> build/lib.linux-x86_64-3.7/osgeo
    copying osgeo/ogr.py -> build/lib.linux-x86_64-3.7/osgeo
    Fixing build/lib.linux-x86_64-3.7/gdal.py build/lib.linux-x86_64-3.7/ogr.py build/lib.linux-x86_64-3.7/osr.py build/lib.linux-x86_64-3.7/gdalconst.py build/lib.linux-x86_64-3.7/gdalnumeric.py build/lib.linux-x86_64-3.7/osgeo/gdal.py build/lib.linux-x86_64-3.7/osgeo/gdalconst.py build/lib.linux-x86_64-3.7/osgeo/osr.py build/lib.linux-x86_64-3.7/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.7/osgeo/gdal_array.py build/lib.linux-x86_64-3.7/osgeo/gnm.py build/lib.linux-x86_64-3.7/osgeo/__init__.py build/lib.linux-x86_64-3.7/osgeo/ogr.py
    Skipping optional fixer: ws_comma
    Fixing build/lib.linux-x86_64-3.7/gdal.py build/lib.linux-x86_64-3.7/ogr.py build/lib.linux-x86_64-3.7/osr.py build/lib.linux-x86_64-3.7/gdalconst.py build/lib.linux-x86_64-3.7/gdalnumeric.py build/lib.linux-x86_64-3.7/osgeo/gdal.py build/lib.linux-x86_64-3.7/osgeo/gdalconst.py build/lib.linux-x86_64-3.7/osgeo/osr.py build/lib.linux-x86_64-3.7/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.7/osgeo/gdal_array.py build/lib.linux-x86_64-3.7/osgeo/gnm.py build/lib.linux-x86_64-3.7/osgeo/__init__.py build/lib.linux-x86_64-3.7/osgeo/ogr.py
    Skipping optional fixer: ws_comma
    running build_ext
    gcc -pthread -B /home/visionosl/anaconda3/envs/dl/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/home/visionosl/anaconda3/envs/dl/include/python3.7m -I/home/visionosl/anaconda3/envs/dl/lib/python3.7/site-packages/numpy/core/include -I/usr/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o
    gcc -pthread -B /home/visionosl/anaconda3/envs/dl/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/home/visionosl/anaconda3/envs/dl/include/python3.7m -I/home/visionosl/anaconda3/envs/dl/lib/python3.7/site-packages/numpy/core/include -I/usr/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o -std=c++11
    building 'osgeo._gdal' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/extensions
    gcc -pthread -B /home/visionosl/anaconda3/envs/dl/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/home/visionosl/anaconda3/envs/dl/include/python3.7m -I/home/visionosl/anaconda3/envs/dl/lib/python3.7/site-packages/numpy/core/include -I/usr/include -c extensions/gdal_wrap.cpp -o build/temp.linux-x86_64-3.7/extensions/gdal_wrap.o -std=c++11 -I/usr/include/gdal
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    extensions/gdal_wrap.cpp:4075:1: error: ‘VSIDIR’ does not name a type
     VSIDIR* wrapper_VSIOpenDir( const char * utf8_path,
     ^
    extensions/gdal_wrap.cpp:4123:38: error: ‘VSIDIR’ was not declared in this scope
     DirEntry* wrapper_VSIGetNextDirEntry(VSIDIR* dir)
                                          ^
    extensions/gdal_wrap.cpp:4123:46: error: ‘dir’ was not declared in this scope
     DirEntry* wrapper_VSIGetNextDirEntry(VSIDIR* dir)
                                                  ^
    extensions/gdal_wrap.cpp:4124:1: error: expected ‘,’ or ‘;’ before ‘{’ token
     {
     ^
    extensions/gdal_wrap.cpp:3225:7: warning: ‘char* get_aligned_buffer(char*, GDALDataType)’ defined but not used [-Wunused-function]
     char* get_aligned_buffer(char* data, GDALDataType ntype)
           ^
    extensions/gdal_wrap.cpp:3244:13: warning: ‘void update_buffer_size(void*, char*, char*, size_t)’ defined but not used [-Wunused-function]
     static void update_buffer_size(void* obj, char* data, char* data_aligned, size_t buf_size)
                 ^
    extensions/gdal_wrap.cpp:3299:5: warning: ‘int GetUseExceptions()’ defined but not used [-Wunused-function]
     int GetUseExceptions() {
         ^
    extensions/gdal_wrap.cpp:3305:6: warning: ‘void UseExceptions()’ defined but not used [-Wunused-function]
     void UseExceptions() {
          ^
    extensions/gdal_wrap.cpp:3322:6: warning: ‘void DontUseExceptions()’ defined but not used [-Wunused-function]
     void DontUseExceptions() {
          ^
    extensions/gdal_wrap.cpp:3370:13: warning: ‘void ClearErrorState()’ defined but not used [-Wunused-function]
     static void ClearErrorState()
                 ^
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/visionosl/anaconda3/envs/dl/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-hosc2bjy/gdal/setup.py'"'"'; __file__='"'"'/tmp/pip-install-hosc2bjy/gdal/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-otn602ug/install-record.txt --single-version-externally-managed --compile --install-headers /home/visionosl/anaconda3/envs/dl/include/python3.7m/gdal Check the logs for full command output.

Also tried installing GDAL before pip install solaris

sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get update
sudo apt-get install gdal-bin
sudo apt-get install libgdal-dev
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
pip install GDAL

results in same gdal error.

nrweir commented 4 years ago

@dudeperf3ct,

It looks like the first issue is a conda error unrelated to solaris (see the related issue opened on conda's GitHub repo here), and the latter two are GDAL install errors which are, unfortunately, very common (to the point of solving them almost being a rite of passage). I'd recommend you try installing a slightly older version of conda and re-trying the conda install. Closing this issue since it's not directly solaris-related.

dudeperf3ct commented 4 years ago

Hey @nrweir, I tried downgrading conda to version 4.6 but still gdal error persists. What version of conda is recommended?

If we can't install solaris by conda and pip, is there any other way?

nrweir commented 4 years ago

Conda's current release is at 4.8.2 - I'm currently running conda 4.7.12 and haven't had the problems you indicated.

I'm not sure it's accurate to say that solaris can't be installed by conda/pip - I believe you need to resolve your conda/GDAL install issues, which aren't really related to this project, at which point you'll be good to go. I can sympathize with the challenge of getting that working, though. Best of luck!

CosmiQ / solaris