Update GDAL

[Zeitsperre]

Attempting to run an update with the newer version (GDAL-2.3.1) to see if it builds properly. If not, I'll default to GDAL-2.2.4.

[Zeitsperre]

For more information: https://anaconda.org/conda-forge/gdal/files

[cehbrecht]

@Zeitsperre You don't need to update the env-export.yml. We can do this when we have a working conda env again. I would try locally to get the conda env working again. You need to remove some of the version pins ... and probably we need to drop some dependencies.

[Zeitsperre]

@dhuard This latest push seems to install fine except for an issue with icclim:

/home/tjs/.conda/envs/flyingpigeon-dev/bin/python /home/tjs/git/flyingpigeon/flyingpigeon/processes/wps_ocgis_func.py
Traceback (most recent call last):
  File "/home/tjs/git/flyingpigeon/flyingpigeon/processes/wps_ocgis_func.py", line 23, in <module>
    from flyingpigeon.utils import archiveextract
  File "/home/tjs/git/flyingpigeon/flyingpigeon/__init__.py", line 1, in <module>
    from .wsgi import application
  File "/home/tjs/git/flyingpigeon/flyingpigeon/wsgi.py", line 4, in <module>
    from .processes import processes
  File "/home/tjs/git/flyingpigeon/flyingpigeon/processes/__init__.py", line 66, in <module>
    ] + [c() for c in OCGIS_INDEX_PROCESSES]
  File "/home/tjs/git/flyingpigeon/flyingpigeon/processes/wps_ocgis_func.py", line 95, in __init__
    self.load_meta()
  File "/home/tjs/git/flyingpigeon/flyingpigeon/processes/wps_ocgis_func.py", line 110, in load_meta
    self.ocgis_cls = fr[self.key]
KeyError: 'freezethaw'

Since this function is pulling icclim from ocgis, is it possible that ocgis has changed the keyword for this indicator? Any advice is appreciated.

[huard]

Hum, weird. No, nothing has changed on the icclim side. If you output the content of the function registry (fr), what do you see ?

[huard]

Tested on my end. Can't reproduce icclim issue.

[Zeitsperre]

@huard Is it building for you?

[huard]

Make install then pytest ran without a hitch on ubuntu 16.04. Mind you my dev environment is a bit cluttered, so this is not exactly a clean slate.

[Zeitsperre]

I think the issue might be related to libnetcdf: It can't seem to find the source file and therefiore doesn't see the API. I'm going to downgrade the netcdf to see if that fixes it. I've fixed this manually on my laptop by creating symlinks to source files but that's a really "hackjob" way of doing things I'd rather not propagate.

[Zeitsperre]

Making some headway on this. My suspicions were correct about libnetcdf not being found. This is a known bug with Linux systems for modern libnetcdf installs. The fix isn't here yet but maybe we can add this workaround into the environment.yml as a bash command to run at the end. The workaround is as follows:

$ cd $HOME/.conda/envs/<conda_env_name>/lib
$ ln -s libnetcdf.so libnetcdf.so.11

The old problem dealing with "KeyError: 'freezethaw' is back though. Traceback from a failed test:

../../.conda/envs/flyingpigeon/lib/python2.7/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
flyingpigeon/__init__.py:1: in <module>
    from .wsgi import application
flyingpigeon/wsgi.py:4: in <module>
    from .processes import processes
flyingpigeon/processes/__init__.py:66: in <module>
    ] + [c() for c in OCGIS_INDEX_PROCESSES]
flyingpigeon/processes/wps_ocgis_func.py:94: in __init__
    self.load_meta()
flyingpigeon/processes/wps_ocgis_func.py:109: in load_meta
    self.ocgis_cls = fr[self.key]
E   KeyError: 'freezethaw'

Still looking into this. Will try different versions of ICCLIM.

[huard]

Unlikely to be related to icclim. freezethaw is a custom ocgis object.

[Zeitsperre]

Looks like it. I don't think the freezethaw object exists prior to ocgis=2.1.0. That's probably the major issue. Continuing down the rabbit hole.

[Zeitsperre]

@cehbrecht I was able to unpin several libraries while still passing (most) build tests. I also migrated from an older ocgis-next to a newer ocgis that includes the freezethaw algorithm.

Due to an installation bug with Conda's libnetcdf4, a workaround is needed to reconnect the source files. the command needed is to run the following within the $HOME/.conda/envs/<conda_env_name>/lib directory:

ln -s libnetcdf.so libnetcdf.so.11

I've read that there are ways to runs shell scripts during/after conda build but I am unfamiliar with how that is done. This workaround is needed until the Conda maintainers fix this. Do you know how this is done?

[nilshempelmann]

@cehbrecht we should merge and work on one branch. I leave it up to you.

[cehbrecht]

@nilshempelmann @Zeitsperre I can't decide how you like to work together :)

I have send an invitation to @Zeitsperre to join the birdhouse organization. You may merge this PR to master (though it is broken) and work together on a dev branch or even on master. Don't invest to much time ... just reduce it to something we can release as 1.2.1. My personal approach would be to just keep the "ocgis" processes enabled ... you could start with a working ocgis environment.

[cehbrecht]

@Zeitsperre @nilshempelmann ... an additional hint :) You should keep pywps 4.0.0 otherwise you would need to fix the workdir settings in the processes. We can do this when we move to the "new" flyingpigeon generated from the cookiecutter.

[Zeitsperre]

Hey all. Sorry to make such a mess. I'll follow @cehbrecht 's advice on this and be fine working in the same branch as @nilshempelmann. I don't have write access so if someone wants to merge it, we can start fixing the build issues in a more concerted way.

@nils do you want to conue building from the 'next' branch?

[nilshempelmann]

PR is merged. next up to master. @Zeitsperre : you should have push rights now. I let you work (its evening in Europe ... ). Propose to use the next branch, just to get used to the habit. I ll wrap up tomorrow. Good luck

[Zeitsperre]

Figured out how to accept the invite so I have write access now. I'll try my hand at it some more. Thanks again!