search

NOAA-EMC / EMC_verif-global

Global Forecast System (GFS) verification package using MET and METplus
8 stars 14 forks source link

Create Modulefile for S4 #126

Closed InnocentSouopgui-NOAA closed 3 months ago

InnocentSouopgui-NOAA commented 3 months ago

Resolve issue #115

This creates a module for S4 and removes module loads from ush/load_modules.sh. This was tested in stand alone mode on S4.

malloryprow commented 3 months ago

I'm unable to test on S4 so I'll trust all is working properly. Is there anything else you need to add to this PR or can I merge?

InnocentSouopgui-NOAA commented 3 months ago

I'm unable to test on S4 so I'll trust all is working properly. Is there anything else you need to add to this PR or can I merge?

I will run few more tests today, and let you know when it's ready. Since we only moved the modules loading to a separate file, I believe it should be enough to check that it is producing the same result as before. That is what I am looking for in my tests.

InnocentSouopgui-NOAA commented 3 months ago

I am having the error below for 4 plots (out of hundreds). The errors shows with or without modulefile. So it is not related to the modifications in this pull request, but I think that it is worth investigating.

Is the error showing on another machine?

The traceback looks identical for the four cases, but the files being read are differents. I am pasting below the four instances

Working on lat-lon error plots for HINDEX sfc
<...>/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/sfc/ctrl/d5_HINDEX_sfc.nc exists
Plotting ctrl
<...>/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/sfc/csr/d5_HINDEX_sfc.nc exists
Plotting csr - ctrl
Traceback (most recent call last):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 15, in __call__
    return self.fn(this._geom, other._geom, *args)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geos.py", line 584, in errcheck_predicate
    raise PredicateError("Failed to evaluate %s" % repr(func))
shapely.errors.PredicateError: Failed to evaluate <_FuncPtr object at 0x7f7b68d7dc40>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<...>/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 616, in <module>
    CF_ax = plot_subplot_data(
  File "<...>/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 245, in plot_subplot_data
    CF_tmp = ax_tmp.contourf(
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 321, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in contourf
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in <listcomp>
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in get_datalim
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in <listcomp>
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/transforms.py", line 2361, in transform_path_non_affine
    return self._a.transform_path_non_affine(path)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 189, in transform_path_non_affine
    geoms = cpatch.path_to_geos(src_path,
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/patch.py", line 184, in path_to_geos
    collection[-1][0].contains(geom.exterior)):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geometry/base.py", line 747, in contains
    return bool(self.impl['contains'](self, other))
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 18, in __call__
    self._check_topology(err, this, other)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/topology.py", line 35, in _check_topology
    raise TopologicalError(
shapely.errors.TopologicalError: The operation 'GEOSContains_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7f7b656e7820>
Working on lat-lon error plots for WEASD sfc
/scratch/users/isouopgui/verif_global_maps2d_s4/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/precip/ctrl/d5_WEASD_sfc.nc exists
Plotting ctrl
/scratch/users/isouopgui/verif_global_maps2d_s4/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/precip/csr/d5_WEASD_sfc.nc exists
Plotting csr - ctrl
/scratch/users/isouopgui/verif_global_maps2d_s4/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/precip/T_ten/d5_WEASD_sfc.nc exists
Plotting T_ten - ctrl
Traceback (most recent call last):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 15, in __call__
    return self.fn(this._geom, other._geom, *args)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geos.py", line 584, in errcheck_predicate
    raise PredicateError("Failed to evaluate %s" % repr(func))
shapely.errors.PredicateError: Failed to evaluate <_FuncPtr object at 0x7fe7b7e85a00>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/users/isouopgui/devel/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 616, in <module>
    CF_ax = plot_subplot_data(
  File "/data/users/isouopgui/devel/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 245, in plot_subplot_data
    CF_tmp = ax_tmp.contourf(
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 321, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in contourf
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in <listcomp>
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in get_datalim
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in <listcomp>
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/transforms.py", line 2361, in transform_path_non_affine
    return self._a.transform_path_non_affine(path)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 189, in transform_path_non_affine
    geoms = cpatch.path_to_geos(src_path,
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/patch.py", line 184, in path_to_geos
    collection[-1][0].contains(geom.exterior)):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geometry/base.py", line 747, in contains
    return bool(self.impl['contains'](self, other))
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 18, in __call__
    self._check_topology(err, this, other)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/topology.py", line 35, in _check_topology
    raise TopologicalError(
shapely.errors.TopologicalError: The operation 'GEOSContains_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7fe79f4e5100>
Working on lat-lon error plots for CAPE sfc
<...>/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/capecin/ctrl/d5_CAPE_sfc.nc exists
Plotting ctrl
<...>/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/capecin/csr/d5_CAPE_sfc.nc exists
Plotting csr - ctrl
<...>/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/capecin/T_ten/d5_CAPE_sfc.nc exists
Plotting T_ten - ctrl
<...>/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/capecin/T_ten_ocean/d5_CAPE_sfc.nc exists
Plotting T_ten_ocean - ctrl
Traceback (most recent call last):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 15, in __call__
    return self.fn(this._geom, other._geom, *args)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geos.py", line 584, in errcheck_predicate
    raise PredicateError("Failed to evaluate %s" % repr(func))
shapely.errors.PredicateError: Failed to evaluate <_FuncPtr object at 0x7fac18de1880>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<...>/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 616, in <module>
    CF_ax = plot_subplot_data(
  File "<...>/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 245, in plot_subplot_data
    CF_tmp = ax_tmp.contourf(
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 321, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in contourf
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in <listcomp>
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in get_datalim
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in <listcomp>
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/transforms.py", line 2361, in transform_path_non_affine
    return self._a.transform_path_non_affine(path)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 189, in transform_path_non_affine
    geoms = cpatch.path_to_geos(src_path,
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/patch.py", line 184, in path_to_geos
    collection[-1][0].contains(geom.exterior)):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geometry/base.py", line 747, in contains
    return bool(self.impl['contains'](self, other))
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 18, in __call__
    self._check_topology(err, this, other)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/topology.py", line 35, in _check_topology
    raise TopologicalError(
shapely.errors.TopologicalError: The operation 'GEOSContains_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7fac146569a0>
Working on lat-lon error plots for TSOIL 100-200cmUGL
/scratch/users/isouopgui/verif_global_maps2d_s4/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/groundsoil/ctrl/d1_TSOIL_100-200cmUGL.nc exists
Plotting ctrl
/scratch/users/isouopgui/verif_global_maps2d_s4/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/groundsoil/csr/d1_TSOIL_100-200cmUGL.nc exists
Plotting csr - ctrl
/scratch/users/isouopgui/verif_global_maps2d_s4/tmp/verif_global.263425/maps2d/metplus_output/make_met_data_by_VALID/series_analysis/model2model/groundsoil/T_ten/d1_TSOIL_100-200cmUGL.nc exists
Plotting T_ten - ctrl
Traceback (most recent call last):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 15, in __call__
    return self.fn(this._geom, other._geom, *args)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geos.py", line 584, in errcheck_predicate
    raise PredicateError("Failed to evaluate %s" % repr(func))
shapely.errors.PredicateError: Failed to evaluate <_FuncPtr object at 0x7fe7b31b6a00>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/users/isouopgui/devel/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 616, in <module>
    CF_ax = plot_subplot_data(
  File "/data/users/isouopgui/devel/EMC_verif-global/ush/plotting_scripts/plot_maps2d_lat_lon_errors.py", line 245, in plot_subplot_data
    CF_tmp = ax_tmp.contourf(
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 321, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in contourf
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 1589, in <listcomp>
    bboxes = [col.get_datalim(self.transData)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in get_datalim
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/collections.py", line 261, in <listcomp>
    paths = [transform.transform_path_non_affine(p) for p in paths]
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/matplotlib/transforms.py", line 2361, in transform_path_non_affine
    return self._a.transform_path_non_affine(path)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py", line 189, in transform_path_non_affine
    geoms = cpatch.path_to_geos(src_path,
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/cartopy/mpl/patch.py", line 184, in path_to_geos
    collection[-1][0].contains(geom.exterior)):
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/geometry/base.py", line 747, in contains
    return bool(self.impl['contains'](self, other))
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/predicates.py", line 18, in __call__
    self._check_topology(err, this, other)
  File "/opt/miniconda/3.8-s4/lib/python3.8/site-packages/shapely/topology.py", line 35, in _check_topology
    raise TopologicalError(
shapely.errors.TopologicalError: The operation 'GEOSContains_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7fe7af0743a0>
malloryprow commented 3 months ago

I've seen the error before but it isn't all that consist.

https://github.com/SciTools/cartopy/issues/1064 https://github.com/SciTools/cartopy/issues/1546

look related

InnocentSouopgui-NOAA commented 3 months ago

@malloryprow, It sure looks related to those you pointed out. It looks like something that can happened randomly depending on Data.

InnocentSouopgui-NOAA commented 3 months ago

@malloryprow, I did a test on 1, 2, 3, and 4 days of data and got the same result as before adding the module file.

malloryprow commented 3 months ago

Good to merge then?

InnocentSouopgui-NOAA commented 3 months ago

Good to merge then?

Yes.