ecoon / ideal-octo-waffle

workflow for ideal-octo-waffle things
Other
3 stars 3 forks source link

Create a conda/python environment to run this code in #1

Open q-rai opened 6 years ago

q-rai commented 6 years ago

The tutorial is for complete installs. To keep things nicely separated, I'm trying to build a conda environment for the project.

It's been a couple of attempts and I figure it makes more sense to track the progress in an issue right with the code, rather than completely separately. I'll document my progress here, starting by pasting the documentation I have of my attempts so far.

q-rai commented 6 years ago

First attempt!

    conda create -n hydro python=3.6 numpy matplotlib shapely
    source activate hydro
    conda install matplotlib
    conda install -c conda-forge fiona
    conda install -c conda-forge rasterio
    conda install shapely
    pip install MeshPy
    conda install scipy
    conda install vtk
    conda install ipykernel

To add the environment to jupyter notebooks, run python -m ipykernel install --user --name hydro --display-name "Python 3 (Hydro)"

So far, so good. But gdal refused to install. The solution is supposedly this, according to one of many search results I went through:

conda install -c conda-forge fiona "libgdal<2.0"

But this would uninstall rasterio. When trying anyway, I got “An unexpected error has occurred.”

Issues along the way

Result:

    Preparing transaction: done
    Verifying transaction: failed
    # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
    Traceback (most recent call last):``` 
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 819, in __call__
        return func(*args, **kwargs)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/main.py", line 78, in _main
        exit_code = do_call(args, p)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/main_install.py", line 11, in execute
        install(args, parser, 'install')
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 253, in install
        handle_txn(unlink_link_transaction, prefix, args, newenv)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 282, in handle_txn
        unlink_link_transaction.execute()
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 223, in execute
        self.verify()
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/common/io.py", line 46, in decorated
        return f(*args, **kwds)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 210, in verify
        exceptions = self._verify(self.prefix_setups, self.prefix_action_groups)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 481, in _verify
        cls._verify_transaction_level(prefix_setups),
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 476, in <genexpr>
        exceptions = tuple(exc for exc in concatv(
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 334, in _verify_individual_level
        error_result = axn.verify()
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/path_actions.py", line 292, in verify
        source_sha256 = compute_sha256sum(self.source_full_path)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/gateways/disk/read.py", line 75, in compute_sha256sum
        return _digest_path('sha256', file_full_path)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/gateways/disk/read.py", line 64, in _digest_path
        with open(path, "rb") as fh:
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/common/compat.py", line 121, in open
        errors=errors, newline=newline, closefd=closefd)
    IOError: [Errno 13] Permission denied: '/Users/vib/anaconda/pkgs/vtk-6.3.0-py27_1/bin/vtkHashSource-6.3'
q-rai commented 6 years ago

Attempt 2: Install from YML

Ethan added a YML file for the project which he exported from his anaconda root environment. I tried importing it:

conda env create -n hydro -f water_workflow.yml

Install Issues

    ResolvePackageNotFound:
    libgdal==2.2.4=2

Running Issues

With the environment installed, I tried running the workflow example.

ImportError for Poppler

 ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.76.dylib   Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib   Reason: image not found

Searched for libpoppler ll | grep -n "libpoppler" in /Users/vib/anaconda/envs/hydro/lib

Found similar version /Users/vib/anaconda/envs/hydro/lib/libpoppler.76.dylib

Linked the similar version: ln -s libpoppler.78.dylib libpoppler.76.dylib got rid of the error

pending testing once other errors are gone

ImportError for Kea

ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libkea.1.4.7.dylib.   Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib   Reason: image not found

Linked similar library: ln -s libkea.1.4.8.dylib libkea.1.4.7.dylib

Got a new error:

ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libkea.1.4.7.dylib.   Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib   Reason: Incompatible library version: libgdal.20.dylib requires version 1.4.7 or later, but libkea.1.4.7.dylib provides version 1.4.0

— odd, because the installed version is newer than what’s required.. I couldn't really make much sense of this so I'll come back to it later.

Back to Installing

I cycled back to trying to fix things by installs of specific versions. An issue in the conda forge repository suggested: conda install 'poppler<0.66'

Yet another error…

ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.71.dylib
  Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib
  Reason: image not found

The comments also talk about how the default channel’s gdal may work better.

q-rai commented 6 years ago

Attempt 3: Using IOOS

Someone recommended using the IOOS versions of geography libraries in a Continuum IO issue, so I went ahead and tried that:

conda create -n hydro2 python=3.6 numpy matplotlib shapely ipykernel shapely scipy vtk
conda install -c ioos fiona geopandas rasterio
pip install MeshPy
python -m ipykernel install --user --name hydro2 --display-name "Python 3 (Hydro2)"

And again, the dreaded

ImportError: dlopen(/Users/vib/anaconda/envs/hydro2/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.76.dylib
  Referenced from: /Users/vib/anaconda/envs/hydro2/lib/libgdal.20.dylib
  Reason: image not found

Tried conda install 'poppler<0.66'

And got the other dreaded

ImportError: dlopen(/Users/vib/anaconda/envs/hydro2/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.71.dylib
  Referenced from: /Users/vib/anaconda/envs/hydro2/lib/libgdal.20.dylib
  Reason: image not found
ecoon commented 5 years ago

Ok, I believe I have a reasonable solution. conda-forge appears to be caught up with reasonable versions now.

Note this is with anaconda 5.3.0, and requires a .condarc with conda-forge PRECEEDING defaults (the first line below this ensures this is the case.

$> conda config --prepend channels conda-forge
$> conda create -n ideal_octo_waffle fiona attrs rasterio matplotlib scipy nb_conda pyproj requests
$> conda activate ideal_octo_waffle
$> python -c "import fiona; print(fiona.__version__); import rasterio; print(rasterio.__version__)"
1.8a3
1.0.8

Updating the README now