NCL Shapefile bug (check if fixed each release)

[alaniwi]

The latest build of ncl (ncl=6.6.2=h087f628_23) has a problem that is exposed when trying to plot the London boundary in the following test code:

begin

  wks = gsn_open_wks("x11", "test")

  res              = True
  res@gsnFrame     = False
  res@mpMinLatF    = 51.10
  res@mpMaxLatF    = 51.85
  res@mpMinLonF    = -0.70
  res@mpMaxLonF    = 0.60

  plot = gsn_csm_map(wks, res)

  filename = "england_gor_2011.shp"
  dum1 = gsn_add_shapefile_polylines(wks, plot, filename, True)
  draw(plot)
  frame(wks)

end

Input data for test: england_gor_2011.tar.gz

With a working release of ncl, you get:

with a broken release, you get:

I have tested which releases have the error (not every release is tested because of the time that it takes, but after initial experimentation with a couple of recent releases, I then did a binary search to identify the first broken release, on the assumption that the breakage was introduced at one point and then persisted thereafter). This identified that build h8482486_9 appears to be the last working release.

ncl=6.6.2=h9ce15c7_1 OK
ncl=6.6.2=h7290063_6 OK
ncl=6.6.2=h8482486_9 OK
ncl=6.6.2=had3379d_10 broken
ncl=6.6.2=h95b68c9_11 broken
ncl=6.6.2=h297df27_19 broken  <=== release in latest Jaspy environment
ncl=6.6.2=h0f0234b_20 broken
ncl=6.6.2=h087f628_23 broken  <=== latest available release on conda-forge

For the next Jaspy environment, we should see if a later release on conda-forge fixes the problem. If so, we should upgrade to it. If not, we should downgrade to h8482486_9.

[alaniwi]

We are going to have to close this as WONTFIX. It seems that an upgrade is not currently available, and a sufficient downgrade is not possible in the same environment as everything else that we need to include in Jaspy.

Why an NCL downgrade is not possible

The following issue is encountered:

• Downgrading ncl will require downgrading gdal.
• If separately fiona is required in the environment, then the earlier gdal precludes installation of the latest fiona
• Attempts to install earlier version of fiona fail because it depends on python 2.7.

Specifically, fixing only the ncl release (ncl=6.6.2=h8482486_9) in initial.yml, while otherwise using >= dependencies including fiona >= 1.8.13, results in:

Problem: package fiona-1.8.13-py27h900e953_0 requires python >=2.7,<2.8.0a0, but none of the providers can be installed

When building an environment containing the fiona >= 1.8.13 constraint, and not at the same time attempting to downgrade ncl, the fiona version that actually gets installed is 1.8.18. If fiona>=1.8.18 is imposed explicitly as an additional constraint on the environment that also requires ncl=6.6.2=h8482486_9, in order to see what else breaks, then this results in:

Problem: package fiona-1.8.18-py37h527b4ca_0 requires libgdal >=3.1.4,<3.2.0a0, but none of the providers can be installed

It might be possible to resolve this by finding some intermediate version of fiona (compatible with both an earlier gdal and also Python 3), but even without this issue, we do not want to be downgrading gdal just for sake of an issue with NCL.

Potential workaround for users

Users requiring an environment containing an earlier version of ncl could build a separate environment of their own, which does not also contain the same range of other packages that we are trying to provide in Jaspy. A simple recipe for this, after ensuring that conda is in the path (e.g. after installing miniconda into a user directory) would be:

conda create -n my_ncl_env
conda activate my_ncl_env    # or "source activate ...."
conda install ncl=6.6.2=h8482486_9

Optionally if conda install mamba is done (in the base environment before activating my_ncl_env), then the last conda install can be replaced with the more efficient mamba install.

[agstephens]

Resolution: use latest ncl but tell users about the issue.

[agstephens]

This issue has been tagged as a known problem in the JASMIN Help pages:

https://help.jasmin.ac.uk/article/4729-jaspy-envs

[alaniwi]

Jaspy autumn release has h16ac178_24 and still has this issue.

Given that we are not fixing, and have documented it already, do we need this ticket still open for anything @agstephens ?

[alaniwi]

A possible temporary workaround is mentioned at https://github.com/NCAR/ncl/issues/176

Unfortunately this is not fixing it for us. The file being patched (shapefile_utils.ncl) does not exist in our installation. Our test code calls gsn_open_wks, which is contained in the file $CONDA_PREFIX/lib/ncarg/nclscripts/csm/gsn_code.ncl and which contains some similar lines, namely

   lon = f->x
   lat = f->y

in three places. Applying a similar fix in each of these places (i.e. swap lon and lat) still does not lead to the outline of London being plotted in our test case.

[alaniwi]

Here is what the patch used in the workaround does (difference between original and patched version of the shapefile_utils.ncl file in question):

--- shapefile_utils.ncl 2019-01-11 21:01:09.000000000 +0000
+++ shapefile_utils.ncl.txt 2021-10-31 20:35:57.000000000 +0000
@@ -1,3 +1,7 @@
+; *** SHAPEFILE_UTILS.NCL ***
+; *** TEMPORARY PATCHED VERSION, SWAP X AND Y COORDINATES ***
+; *** Patched 2021 Oct 31 by Dave Allured, NOAA/PSL/CIRES ***
+
 ;======================================================================
 ; This script contains a number of functions and procedures for
 ; working with shapefiles in NCL.
@@ -45,8 +49,13 @@
   var_names    = getfilevarnames(f)                  ; variable names
   geo_dims     = getfilevardimsizes(f,"geometry")    ; features on file
   num_features = geo_dims(0)
-  lon          = f->x
-  lat          = f->y
+
+;;;  lon          = f->x
+;;;  lat          = f->y
+
+     lat          = f->x   ; TEMPORARY WORKAROUND FOR SWAPPED COORDINATES
+     lon          = f->y   ; due to library change, probably GDAL version 3
+
   nvars        = dimsizes(var_names)

   print("======================================================================")
@@ -237,9 +246,13 @@
   numFeatures   = geomDims(0)

 ;---Read shapefile lat/lon
-  lon = f->x
-  lat = f->y
-;
+
+;;;  lon = f->x
+;;;  lat = f->y
+
+     lat = f->x        ; TEMPORARY WORKAROUND FOR SWAPPED COORDINATES
+     lon = f->y        ; due to library change, probably GDAL version 3
+
 ; If shp_var_name is specified, then get the approximate lat/lon box that 
 ; encloses the shapefile areas of interest. This can make the 
 ; gc_inout checking go faster later. If the user has input
@@ -527,8 +540,13 @@
   wks = gsn_open_wks("png",get_shapefile_prefix(shapefile_name))

 ;---Read lat/lon and create map
-  lat = f->y
-  lon = f->x
+
+;;;  lat = f->y
+;;;  lon = f->x
+
+     lat = f->x        ; TEMPORARY WORKAROUND FOR SWAPPED COORDINATES
+     lon = f->y        ; due to library change, probably GDAL version 3
+
   map = create_shapefile_map(wks,shapefile_name,geometry_type,lat,lon)

   sres = True

[alaniwi]

Version 6.6.2, to be installed via jasmin-sci for the time being, fixes this issue. Assuming that we will not downgrade when we are ready to readd it to Jaspy, we can close this.