dbekaert / RAiDER

Raytracing Atmospheric Delay Estimation for RADAR
Apache License 2.0
70 stars 39 forks source link

[BUG] Python crash in test/test_interpolator.py #327

Open EJFielding opened 3 years ago

EJFielding commented 3 years ago

Describe the bug Test program test/test_interpolator.py crashes Python

To Reproduce Steps to reproduce the behavior:

  1. py.test test/
  2. output:
    
    ============================================================================ test session starts ============================================================================
    platform darwin -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
    rootdir: /Users/fielding/tools/RAiDER
    collected 258 items

test/test_checkArgs.py ................................................ [ 18%] test/test_chunking.py ......... [ 22%] test/test_delay.py . [ 22%] test/test_dem.py ......... [ 25%] test/test_gnss.py .... [ 27%] test/test_hdf5_parallel.py s [ 27%] test/test_integration.py ... [ 29%] test/test_interpolator.py ...Fatal Python error: Aborted

Thread 0x00007000186d4000 (most recent call first): File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/threading.py", line 300 in wait File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/threading.py", line 552 in wait File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/tqdm/_monitor.py", line 60 in run File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x000000010da53dc0 (most recent call first): File "/Users/fielding/tools/RAiDER/test/test_interpolator.py", line 69 in test_interpolate_along_axis File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 255 in File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/config/init.py", line 163 in main File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/_pytest/config/init.py", line 185 in console_main File "/Users/fielding/anaconda3/envs/RAiDER/bin/py.test", line 11 in Abort


**Expected behavior**
I expected the test to run and at least provide Python traceback instead of crash.

**Screenshots**
Not applicable.

**Desktop (please complete the following information):**
 - RAiDER git tag: [e.g. git show --summary]  

commit bb43a9e2ea70838ccaed3ed03087c172810f1591 (HEAD -> dev, origin/dev, origin/HEAD) Merge: aabfd0c 4e2ebda Author: Jeremy Maurer maurer.jeremy@gmail.com Date: Tue Sep 21 11:05:21 2021 -0500

Merge pull request #320 from jlmaurer/add_rest_model_levels

Add fixed zlevels to other weather models ie non ECMWF

 - OS: MacOS Catalina, conda installation.

**Additional context**
Add any other context about the problem her
EJFielding commented 3 years ago

My other Mac MacOS Catalina does not crash when running the tests. I did the same Conda installation on both. Very strange.

=============================================== test session starts ================================================
platform darwin -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/fielding/tools/RAiDER
collected 258 items

test/test_checkArgs.py ................................................                                      [ 18%]
test/test_chunking.py .........                                                                              [ 22%]
test/test_delay.py .                                                                                         [ 22%]
test/test_dem.py .........                                                                                   [ 25%]
test/test_gnss.py ....                                                                                       [ 27%]
test/test_hdf5_parallel.py s                                                                                 [ 27%]
test/test_integration.py ...                                                                                 [ 29%]
test/test_interpolator.py ............................................                                       [ 46%]
test/test_llreader.py ......                                                                                 [ 48%]
test/test_losreader.py ........................                                                              [ 57%]
test/test_scenario_1.py F...sF                                                                               [ 60%]
test/test_scenario_2.py F                                                                                    [ 60%]
test/test_util.py ................s...............                                                           [ 72%]
test/cli/test_argument_parsers.py ....                                                                       [ 74%]
test/cli/test_validators.py .............................................                                    [ 91%]
test/weather_model/test_downloaders.py ...                                                                   [ 93%]
test/weather_model/test_processWM.py ..                                                                      [ 93%]
test/weather_model/test_weather_model.py ................                                                    [100%]

===================================================== FAILURES =====================================================
______________________________________________ test_tropo_delay_ERAI _______________________________________________

tmp_path = PosixPath('/private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_tropo_delay_ERAI0')

    @pytest.mark.long
    def test_tropo_delay_ERAI(tmp_path):
        '''
        Scenario:
        1: Small area, ERAI, Zenith delay
        '''
>       core_test_tropo_delay(tmp_path, modelName="ERAI")

/Users/fielding/tools/RAiDER/test/test_scenario_1.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/fielding/tools/RAiDER/test/test_scenario_1.py:134: in core_test_tropo_delay
    (_, _) = tropo_delay(args)
/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/delay.py:121: in tropo_delay
    makePlots=True
/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/processWM.py:88: in prepareWeatherModel
    zref=zref,
/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/models/weatherModel.py:201: in load
    self.load_weather(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <RAiDER.models.erai.ERAI object at 0x7fe1595e2f10>, args = (), kwargs = {}

    def load_weather(self, *args, **kwargs):
        '''
        Consistent class method to be implemented across all weather model types.
        As a result of calling this method, all of the variables (x, y, z, p, q,
        t, wet_refractivity, hydrostatic refractivity, e) should be fully
        populated.
        '''
>       self._load_model_level(*self.files)
E       AttributeError: 'ERAI' object has no attribute '_load_model_level'

/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/models/ecmwf.py:69: AttributeError
----------------------------------------------- Captured stdout call -----------------------------------------------
Weather model ERA-I is available from 1979-01-01 00:00:00-2019-08-31 00:00:00
2021-10-02 20:35:57 ECMWF API python library 1.6.1
2021-10-02 20:35:57 ECMWF API at https://api.ecmwf.int/v1
2021-10-02 20:35:58 Welcome Eric Fielding
WARNING: Query point bounds are 15.35/18.65/-103.65/-99.35
WARNING: Query time: 2018-01-03 23:00:00
ERROR: "ecmwf.API error 1: User 'ericfielding' has not access to datasets/interim. Please accept the terms and conditions at http://apps.ecmwf.int/datasets/licences/general"
Traceback (most recent call last):
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/models/ecmwf.py", line 150, in _fetch
    out
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/models/ecmwf.py", line 192, in _get_from_ecmwf
    "target": out,    # target: the name of the output file.
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 594, in retrieve
    verbose=self.verbose,
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 418, in __init__
    info = self.connection.call("%s/%s/%s" % (self.url, self.service, "info")).get(
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 148, in wrapped
    return func(self, *args, **kwargs)
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 350, in call
    raise APIException("ecmwf.API error 1: %s" % (self.last["error"],))
ecmwfapi.api.APIException: "ecmwf.API error 1: User 'ericfielding' has not access to datasets/interim. Please accept the terms and conditions at http://apps.ecmwf.int/datasets/licences/general"
------------------------------------------------ Captured log call -------------------------------------------------
DEBUG    RAiDER:delay.py:100 Starting to run the weather model calculation
DEBUG    RAiDER:delay.py:101 Time type: <class 'datetime.datetime'>
DEBUG    RAiDER:delay.py:102 Time: 20180103
DEBUG    RAiDER:delay.py:103 Flag type is files
DEBUG    RAiDER:delay.py:104 DEM/height type is "dem"
DEBUG    RAiDER:delay.py:110 Beginning weather model pre-processing
DEBUG    RAiDER:delay.py:111 Download-only is False
INFO     RAiDER:weatherModel.py:247 Weather model ERA-I is available from 1979-01-01 00:00:00-2019-08-31 00:00:00
WARNING  RAiDER:ecmwf.py:153 Query point bounds are 15.35/18.65/-103.65/-99.35
WARNING  RAiDER:ecmwf.py:154 Query time: 2018-01-03 23:00:00
ERROR    RAiDER:ecmwf.py:155 "ecmwf.API error 1: User 'ericfielding' has not access to datasets/interim. Please accept the terms and conditions at http://apps.ecmwf.int/datasets/licences/general"
Traceback (most recent call last):
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/models/ecmwf.py", line 150, in _fetch
    out
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/RAiDER-0.0.1-py3.7-macosx-10.9-x86_64.egg/RAiDER/models/ecmwf.py", line 192, in _get_from_ecmwf
    "target": out,    # target: the name of the output file.
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 594, in retrieve
    verbose=self.verbose,
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 418, in __init__
    info = self.connection.call("%s/%s/%s" % (self.url, self.service, "info")).get(
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 148, in wrapped
    return func(self, *args, **kwargs)
  File "/Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/ecmwfapi/api.py", line 350, in call
    raise APIException("ecmwf.API error 1: %s" % (self.last["error"],))
ecmwfapi.api.APIException: "ecmwf.API error 1: User 'ericfielding' has not access to datasets/interim. Please accept the terms and conditions at http://apps.ecmwf.int/datasets/licences/general"
______________________________________________ test_tropo_delay_GMAO _______________________________________________

tmp_path = PosixPath('/private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_tropo_delay_GMAO0')

    @pytest.mark.long
    def test_tropo_delay_GMAO(tmp_path):
        '''
        Scenario:
        1: Small area, GMAO, Zenith delay
        '''
>       core_test_tropo_delay(tmp_path, modelName="GMAO")

/Users/fielding/tools/RAiDER/test/test_scenario_1.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

tmp_path = PosixPath('/private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_tropo_delay_GMAO0')
modelName = 'GMAO'

    def core_test_tropo_delay(tmp_path, modelName):
        '''
        Scenario:
        1: Small area, Zenith delay
        '''
        lats = gdal_open(os.path.join(
            SCENARIO_DIR, 'geom', 'lat.dat'
        ))
        lons = gdal_open(os.path.join(
            SCENARIO_DIR, 'geom', 'lon.dat'
        ))

        if modelName == 'ERAI':
            time = datetime.datetime(2018, 1, 3, 23, 0)
        elif modelName == 'NCMR':
            time = datetime.datetime(2018, 7, 1, 0, 0)
        else:
            time = datetime.datetime(2020, 1, 3, 23, 0)

        wmLoc = os.path.join(SCENARIO_DIR, 'weather_files')
        if not os.path.exists(wmLoc):
            os.mkdir(wmLoc)

        _, model_obj = modelName2Module(modelName)
        wet_file, hydro_file = makeDelayFileNames(
            time, Zenith, "envi", modelName, tmp_path
        )

        with pushd(tmp_path):
            # packing the dictionairy
            args = {}
            args['los'] = Zenith
            args['lats'] = lats
            args['lons'] = lons
            args['ll_bounds'] = (15.75, 18.25, -103.24, -99.75)
            args['heights'] = ("dem", os.path.join(TEST_DIR, "test_geom", "warpedDEM.dem"))
            args['pnts_file'] = 'lat_query_points.h5'
            args['flag'] = "files"
            args['weather_model'] = {"type": model_obj(), "files": None, "name": modelName}
            args['wmLoc'] = wmLoc
            args['zref'] = 20000.
            args['outformat'] = "envi"
            args['times'] = time
            args['out'] = tmp_path
            args['download_only'] = False
            args['wetFilenames'] = wet_file
            args['hydroFilenames'] = hydro_file
            args['verbose'] = True

            (_, _) = tropo_delay(args)

            # get the results
            wet = gdal_open(wet_file)
            hydro = gdal_open(hydro_file)
            true_wet = gdal_open(
                os.path.join(
                    SCENARIO_DIR,
                    modelName + "/wet.envi"
                ),
                userNDV=0.
            )
            true_hydro = gdal_open(
                os.path.join(
                    SCENARIO_DIR,
                    modelName + "/hydro.envi"
                ),
                userNDV=0.
            )

            # get the true delay from the weather model
>           assert np.nanmax(np.abs((wet - true_wet) / true_wet)) < _RTOL
E           AssertionError: assert 0.012336466 < 0.01
E            +  where 0.012336466 = <function nanmax at 0x7fe1c8317440>(array([[       nan,        nan,        nan,        nan,        nan,\n               nan,        nan,        nan,       ...141, 0.00628924, 0.00521209,\n        0.00344014,        nan,        nan, 0.00389785, 0.00502395]],\n      dtype=float32))
E            +    where <function nanmax at 0x7fe1c8317440> = np.nanmax
E            +    and   array([[       nan,        nan,        nan,        nan,        nan,\n               nan,        nan,        nan,       ...141, 0.00628924, 0.00521209,\n        0.00344014,        nan,        nan, 0.00389785, 0.00502395]],\n      dtype=float32) = <ufunc 'absolute'>(((array([[       nan,        nan,        nan,        nan,        nan,\n               nan,        nan,        nan,       ...486, 0.12377633, 0.1724519 ,\n        0.17415401, 0.16796733, 0.15622063, 0.09607986, 0.09343264]],\n      dtype=float32) - array([[       nan,        nan,        nan,        nan,        nan,\n               nan,        nan,        nan,       ...807, 0.12300274, 0.17155772,\n        0.17355695,        nan,        nan, 0.09570681, 0.09296558]],\n      dtype=float32)) / array([[       nan,        nan,        nan,        nan,        nan,\n               nan,        nan,        nan,       ...807, 0.12300274, 0.17155772,\n        0.17355695,        nan,        nan, 0.09570681, 0.09296558]],\n      dtype=float32)))
E            +      where <ufunc 'absolute'> = np.abs

/Users/fielding/tools/RAiDER/test/test_scenario_1.py:155: AssertionError
----------------------------------------------- Captured stdout call -----------------------------------------------
WARNING: Weather model already exists, please remove it ("['/Users/fielding/tools/RAiDER/test/scenario_1/weather_files/GMAO_2020_01_03_T23_00_00.nc']") if you want to download a new one.
Extent of the weather model is (xmin, ymin, xmax, ymax):-104.22, 15.12, -99.22, 18.88
Extent of the input is (xmin, ymin, xmax, ymax): -103.25, 15.75, -99.75, 18.25
WARNING: The processed weather model file already exists, so I will use that.
Finished writing data to /private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_tropo_delay_GMAO0/GMAO_wet_20200103T230000_std.envi
------------------------------------------------ Captured log call -------------------------------------------------
DEBUG    RAiDER:delay.py:100 Starting to run the weather model calculation
DEBUG    RAiDER:delay.py:101 Time type: <class 'datetime.datetime'>
DEBUG    RAiDER:delay.py:102 Time: 20200103
DEBUG    RAiDER:delay.py:103 Flag type is files
DEBUG    RAiDER:delay.py:104 DEM/height type is "dem"
DEBUG    RAiDER:delay.py:110 Beginning weather model pre-processing
DEBUG    RAiDER:delay.py:111 Download-only is False
WARNING  RAiDER:processWM.py:52 Weather model already exists, please remove it ("['/Users/fielding/tools/RAiDER/test/scenario_1/weather_files/GMAO_2020_01_03_T23_00_00.nc']") if you want to download a new one.
INFO     RAiDER:weatherModel.py:447 Extent of the weather model is (xmin, ymin, xmax, ymax):-104.22, 15.12, -99.22, 18.88
INFO     RAiDER:weatherModel.py:449 Extent of the input is (xmin, ymin, xmax, ymax): -103.25, 15.75, -99.75, 18.25
WARNING  RAiDER:processWM.py:92 The processed weather model file already exists, so I will use that.
DEBUG    RAiDER:delay.py:133 Beginning DEM calculation
DEBUG    RAiDER:delay.py:137 DEM height range for the queried region is -5.29-2259.36 m
DEBUG    RAiDER:delay.py:146 Lats shape is (11, 15)
DEBUG    RAiDER:delay.py:149 lat/lon box is 15.750000/18.250000/-103.250000/-99.750000 (SNWE)
DEBUG    RAiDER:delay.py:170 Beginning line-of-sight calculation
DEBUG    RAiDER:delay.py:40 Beginning delay calculation
DEBUG    RAiDER:delay.py:41 Max integration height is 20000.0 m
DEBUG    RAiDER:delay.py:42 Reference integration step is 15.0 m
DEBUG    RAiDER:delay.py:71 Finished delay calculation
INFO     RAiDER:delay.py:214 Finished writing data to /private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_tropo_delay_GMAO0/GMAO_wet_20200103T230000_std.envi
DEBUG    RAiDER:utilFcns.py:129 Using user-supplied NoDataValue
DEBUG    RAiDER:utilFcns.py:129 Using user-supplied NoDataValue
________________________________________________ test_computeDelay _________________________________________________

tmp_path = PosixPath('/private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_computeDelay0')

    @pytest.mark.long
    def test_computeDelay(tmp_path):
        '''
        Scenario to use:
        2: GNSS station list
        '''
        wetName = 'stations_with_Delays.csv'
        wetFile = os.path.join(SCENARIO_DIR, wetName)
        # Not used for station file input, only passed for consistent input arguments
        hydroFile = wetFile

        # load the weather model type and date for the given scenario
        wmLoc = os.path.join(SCENARIO_DIR, 'weather_files')

        true_delay = os.path.join(SCENARIO_DIR, 'ERA5_true_GNSS.csv')

        station_file = os.path.join(SCENARIO_DIR, 'stations.csv')
        copyfile(station_file, wetFile)
        stats = pd.read_csv(station_file)
        lats = stats['Lat'].values
        lons = stats['Lon'].values

        _, model_obj = modelName2Module('ERA5')

        with pushd(tmp_path):

            # packing the dictionairy
            args = {}
            args['los'] = Zenith
            args['lats'] = lats
            args['lons'] = lons
            args['ll_bounds'] = (33.746, 36.795, -118.312, -114.892)
            args['heights'] = ('merge', [wetFile])
            args['pnts_file'] = 'ERA5_true_GNSS_query_points.h5'
            args['flag'] = "station_file"
            args['weather_model'] = {"type": model_obj(), "files": None, "name": "ERA5"}
            args['wmLoc'] = None
            args['zref'] = 20000.
            args['outformat'] = "csv"
            args['times'] = datetime(2020, 1, 3, 23, 0, 0)
            args['out'] = tmp_path
            args['download_only'] = False
            args['wetFilenames'] = wetFile
            args['hydroFilenames'] = hydroFile
            args['verbose'] = True

            (_, _) = tropo_delay(args)

        # get the results
        est_delay = pd.read_csv(wetFile)
        true_delay = pd.read_csv(true_delay)

        # get the true delay from the weather model
>       assert np.nanmax(np.abs((est_delay['wetDelay'].values - true_delay['wetDelay'].values) / true_delay['wetDelay'].values)) < _RTOL
E       ValueError: operands could not be broadcast together with shapes (9,) (8,)

/Users/fielding/tools/RAiDER/test/test_scenario_2.py:72: ValueError
----------------------------------------------- Captured stdout call -----------------------------------------------
Weather model ERA-5 is available from 1950-01-01 00:00:00-Present
WARNING: A DEM already exists in /Users/fielding/tools/RAiDER/test/scenario_2, checking extents
Finished writing data to /Users/fielding/tools/RAiDER/test/scenario_2/stations_with_Delays.csv
----------------------------------------------- Captured stderr call -----------------------------------------------

------------------------------------------------ Captured log call -------------------------------------------------
DEBUG    RAiDER:delay.py:100 Starting to run the weather model calculation
DEBUG    RAiDER:delay.py:101 Time type: <class 'datetime.datetime'>
DEBUG    RAiDER:delay.py:102 Time: 20200103
DEBUG    RAiDER:delay.py:103 Flag type is station_file
DEBUG    RAiDER:delay.py:104 DEM/height type is "merge"
DEBUG    RAiDER:delay.py:110 Beginning weather model pre-processing
DEBUG    RAiDER:delay.py:111 Download-only is False
INFO     RAiDER:weatherModel.py:247 Weather model ERA-5 is available from 1950-01-01 00:00:00-Present
DEBUG    RAiDER:processWM.py:100 Number of weather model nodes: 9690
DEBUG    RAiDER:processWM.py:104 Shape of weather model: (15, 17, 38)
DEBUG    RAiDER:processWM.py:108 Bounds of the weather model: 33.35/36.85/-118.71/-114.71 (SNWE)
DEBUG    RAiDER:processWM.py:110 Weather model: ERA-5
DEBUG    RAiDER:processWM.py:113 Mean value of the wet refractivity: 6.034212
DEBUG    RAiDER:processWM.py:117 Mean value of the hydrostatic refractivity: 124.192192
DEBUG    RAiDER:processWM.py:119
======Weather Model class object=====
Weather model time: 2020-01-03 23:00:00
Latitude resolution: 0.2
Longitude resolution: 0.2
Native projection: epsg:4326
ZMIN: -100.0
ZMAX: 15000.0
k1 = 0.776
k2 = 0.233
k3 = 3750.0
Humidity type = q
=====================================
Class name: ea
Dataset: era5
=====================================
A: []
B: []
Number of points in Lon/Lat = 15/17
Total number of grid points (3D): 9690
=====================================

DEBUG    RAiDER:delay.py:133 Beginning DEM calculation
WARNING  RAiDER:dem.py:117 A DEM already exists in /Users/fielding/tools/RAiDER/test/scenario_2, checking extents
DEBUG    RAiDER:delay.py:137 DEM height range for the queried region is 0.10-0.50 m
DEBUG    RAiDER:delay.py:146 Lats shape is (9,)
DEBUG    RAiDER:delay.py:149 lat/lon box is 33.746000/36.795000/-118.312000/-114.892000 (SNWE)
DEBUG    RAiDER:delay.py:170 Beginning line-of-sight calculation
DEBUG    RAiDER:delay.py:40 Beginning delay calculation
DEBUG    RAiDER:delay.py:41 Max integration height is 20000.0 m
DEBUG    RAiDER:delay.py:42 Reference integration step is 15.0 m
DEBUG    RAiDER:delay.py:71 Finished delay calculation
INFO     RAiDER:delay.py:214 Finished writing data to /Users/fielding/tools/RAiDER/test/scenario_2/stations_with_Delays.csv
================================================= warnings summary =================================================
test/test_dem.py:77
  /Users/fielding/tools/RAiDER/test/test_dem.py:77: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_1.py:22
  /Users/fielding/tools/RAiDER/test/test_scenario_1.py:22: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_1.py:31
  /Users/fielding/tools/RAiDER/test/test_scenario_1.py:31: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_1.py:40
  /Users/fielding/tools/RAiDER/test/test_scenario_1.py:40: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_1.py:49
  /Users/fielding/tools/RAiDER/test/test_scenario_1.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_1.py:58
  /Users/fielding/tools/RAiDER/test/test_scenario_1.py:58: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_1.py:76
  /Users/fielding/tools/RAiDER/test/test_scenario_1.py:76: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_scenario_2.py:19
  /Users/fielding/tools/RAiDER/test/test_scenario_2.py:19: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

../../anaconda3/envs/RAiDER/lib/python3.7/site-packages/pydap/responses/das.py:14
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/pydap/responses/das.py:14: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Iterable

test/weather_model/test_downloaders.py:14
  /Users/fielding/tools/RAiDER/test/weather_model/test_downloaders.py:14: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/weather_model/test_downloaders.py:25
  /Users/fielding/tools/RAiDER/test/weather_model/test_downloaders.py:25: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/weather_model/test_downloaders.py:36
  /Users/fielding/tools/RAiDER/test/weather_model/test_downloaders.py:36: PytestUnknownMarkWarning: Unknown pytest.mark.long - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.long

test/test_dem.py::test_getDEM
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/dem_stitcher/stitcher.py:125: UserWarning: /private/var/folders/92/tx_tbh3d05x_gpx6ydy5nmd94bbnb9/T/pytest-of-fielding/pytest-3/test_getDEM0 exists; data will be overwritten
    warnings.warn(f'{dest_dir} exists; data will be overwritten')

test/test_dem.py: 64800 warnings
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/geopandas/_vectorized.py:142: DeprecationWarning: An exception was ignored while fetching the attribute `__array_interface__` from an object of type 'Polygon'.  With the exception of `AttributeError` NumPy will always raise this exception in the future.  Raise this deprecation warning to see the original exception. (Warning added NumPy 1.21)
    aout[:] = out

test/test_scenario_1.py::test_tropo_delay_ERA5
test/test_scenario_1.py::test_tropo_delay_ERA5T
test/test_scenario_1.py::test_tropo_delay_MERRA2
test/test_scenario_1.py::test_tropo_delay_GMAO
test/weather_model/test_processWM.py::test_checkContainment
test/weather_model/test_processWM.py::test_checkContainment2
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/rasterio/__init__.py:220: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix be returned.
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

test/test_scenario_2.py::test_computeDelay
test/weather_model/test_downloaders.py::test_era5
test/weather_model/test_downloaders.py::test_era5t
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'cds.climate.copernicus.eu'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
    InsecureRequestWarning,

test/test_scenario_2.py::test_computeDelay
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'download-0000.copernicus-climate.eu'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
    InsecureRequestWarning,

test/weather_model/test_downloaders.py::test_era5
test/weather_model/test_downloaders.py::test_era5t
  /Users/fielding/anaconda3/envs/RAiDER/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'download-0007.copernicus-climate.eu'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
    InsecureRequestWarning,

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================= short test summary info ==============================================
FAILED test/test_scenario_1.py::test_tropo_delay_ERAI - AttributeError: 'ERAI' object has no attribute '_load_mod...
FAILED test/test_scenario_1.py::test_tropo_delay_GMAO - AssertionError: assert 0.012336466 < 0.01
FAILED test/test_scenario_2.py::test_computeDelay - ValueError: operands could not be broadcast together with sha...
============================ 3 failed, 252 passed, 3 skipped, 64825 warnings in 52.77s =============================
EJFielding commented 3 years ago

I found this issue is likely to be due to a bug in parallel threads on MacOS for Python: https://bugs.python.org/issue30385

The suggestion in the post linked above is to add os.environ[“no_proxy”]=“*” to the Python code before it calls any network connections. I tried adding this right before the RAiDER/test/test_interpolator.py", line 69 in test_interpolate_along_axis but it did not help.