.. image:: https://github.com/pacificclimate/pdp_util/workflows/Python%20CI/badge.svg :target: https://github.com/pacificclimate/pdp_util
.. image:: https://github.com/pacificclimate/pdp_util/workflows/Pypi%20Publishing/badge.svg :target: https://github.com/pacificclimate/pdp_util
Python package pdp_util
provides a variety of utility modules that are required to run the PCIC Data Portal <http://www.pacificclimate.org/data>
. In practice, it contains a large part of the non-Pydap <http://www.pydap.org>
code for the PCIC Data Portal [#non-pcic], while the pdp
package pulls everything together in a single application. The utilies can be subdivided into those that support the "pcds" [#pcds] portal, those that support the "raster" portals, and those that serve both (i.e. globally applicable utilities).
agg
module provides aggregation utilities to translate a single HTTP request into multiple OPeNDAP requests, returning a single response.counts
module provides two WSGI applications for providing counts/estimates of the number of stations/observations (respectively) in a group of PCDS stations.filters
module provides error checking and validation for HTTP POST variables describing PCDS station filters.legend
module provides a WSGI application that creates colored legend symbols for PCDS stations.pcds_dispatch
module provides a WSGI application which simulates a directory tree and dispatches requests to various PyDAP-based applications. It provides the basis for our data listings pages.pcds_index
module provides several applications that return various parts of the station listings directory tree.util
module provides a few functions for parsing and validating HTTP POST variables.The ensemble_members
and raster
modules both provide glue applications for communicating with PCIC's metadata database of climate model output (all of which is multi-dimensional "raster" data, otherwise described as being "spatiotemporal" data).
map
module provides a WSGI application which outputs the data selection map along with a variety of user interface controls. It is primarily configured through JavaScript includes.auth
module contains WSGI middleware for providing authentication via OpenID.dbdict
module contains a function to convert dict object into a database DSN.If your workstation has GDAL 3, postgresql, and python 3 and you are on the PCIC VPN, this package can be installed in a virtual environment with Poetry <https://python-poetry.org/>
_ and configured to connect to the replicated database at db3.pcic.uvic.ca by setting the DSN
and PCDS_DSN
environment variables (passwords are available in Team Password Manager) accordingly. This provides a fairly low-overhead way to run tests.
Installation complications
This package currently depends on `pydap-extras 1.0.0 <https://github.com/pacificclimate/pydap-extras>`_, which in turn depends on GDAL 3.0.4.
GDAL 3.0.4 is tricky to install correctly. The following rigmarole, specifically with pre-installations and a special version of `setuptools`, appears to the only way to get a successful installation. A brief explanation follows:
#. GDAL 3.0.4 requires something called `use_2to3`. Modern versions of `setuptools` do not support it; only versions `setuptools<58` do. See, for example,
* https://github.com/nextgis/pygdal/issues/67
* https://github.com/pypa/setuptools/issues/2781
* https://github.com/OSGeo/gdal/issues/7541
We must therefore explicitly install `setuptools<58` before we install `gdal`.
#. GDAL 3.0.4 cannot be installed successfully by later versions of `pip`. Version 22.3.1 does work. We must ensure it is installed before installing `gdal`.
#. GDAL 3.0.4 depends on `numpy`. This is apparently not declared as a dependency but _is_ flagged by `gdal` as a warning if it is not already installed, and causes the installation to fail. The version must be `numpy<=1.21`. Pre-installing `numpy` solves this problem.
#. Poetry somehow still stumbles over installing `gdal==3.0.4` using its own tooling. However, `gdal` can be installed via Poetry into the virtualenv by using the appropriate version of `pip` (see previous item). This circumvents whatever Poetry does.
#. Once the above steps have been taken, the installation can be completed using the normal `poetry install`.
#. Note that dependencies have been organized into groups to make this as simple as possible. If and when later versions of GDAL are specified, this organization and the installation steps are likely to need to be updated. (Possibly, it will become simpler.)
Hence::
# Pre-install initial packages (pip, setuptools, numpy)
poetry install --only initial
# Install gdal using pip3 into the Poetry virtualenv
poetry run pip3 install gdal==3.0.4
# Install rest of project
poetry install
Additional information
Some useful Poetry commands:
poetry install
- installs the package and sets up a virtual environmentpoetry run pytest
- runs pytest, or any other desired command, in the virtual environment in which the package has been installedpoetry show --tree
- see a dependency treepoetry shell
- load the virtual environment in the current terminalexit
- exit the Poetry shell. Don't use deactivate
as you would for a regular python environment, it does not completely exit the Poetry virtualenv.Additional Poetry documentation is available here <https://python-poetry.org/docs/>
_.
.. rubric:: Footnotes
.. [#pcds] Provincial Climate Data Set .. [#non-pcic] Please note, that I can't imagine this package being useful to anyone other than PCIC (aside from the case where someone wants to understand how we run our application). However, if it does happen to be so, you are welcome to use it under the terms outlined in the LICENSE.txt file.