search

openghg / openghg_inversions

University of Bristol Atmospheric Chemistry Research Group RHIME Inversion code (with openghg dependency)
MIT License
5 stars 0 forks source link

Error using weighted algorithm for hfo runs #124

Closed joe-pitt closed 1 month ago

joe-pitt commented 1 month ago

When I run hfo species (currently using inert fps) using the weighted basis function algorithm, I get the following error:

Traceback (most recent call last):
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 77, in optimize_nregions
    return optimize_nregions(bucket, grid, nregion, tol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 77, in optimize_nregions
    return optimize_nregions(bucket, grid, nregion, tol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 77, in optimize_nregions
    return optimize_nregions(bucket, grid, nregion, tol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 970 more times]
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 72, in optimize_nregions
    if get_nregions(bucket, grid) <= nregion + tol and get_nregions(bucket, grid) >= nregion - tol:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 63, in get_nregions
    return np.max(bucket_split_landsea_basis(grid, bucket))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 102, in bucket_split_landsea_basis
    landsea_indices = load_landsea_indices()
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_weighted.py", line 15, in load_landsea_indices
    landsea_indices = xr.open_dataset(Path(__file__).parent / "country-EUROPE-UKMO-landsea-2023.nc")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/api.py", line 573, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 660, in open_dataset
    ds = store_entrypoint.open_dataset(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/store.py", line 58, in open_dataset
    ds = Dataset(vars, attrs=attrs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/dataset.py", line 693, in __init__
    variables, coord_names, dims, indexes, _ = merge_data_and_coords(
                                               ^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/dataset.py", line 422, in merge_data_and_coords
    return merge_core(
           ^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/merge.py", line 699, in merge_core
    collected = collect_variables_and_indexes(aligned, indexes=indexes)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/merge.py", line 362, in collect_variables_and_indexes
    idx, idx_vars = create_default_index_implicit(variable)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/indexes.py", line 1387, in create_default_index_implicit
    index = PandasIndex.from_variables(dim_var, options={})
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/indexes.py", line 642, in from_variables
    obj = cls(data, dim, coord_dtype=var.dtype)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/indexes.py", line 582, in __init__
    index = safe_cast_to_index(array).copy()
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/indexes.py", line 469, in safe_cast_to_index
    index = pd.Index(np.asarray(array), **kwargs)
                     ^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/indexing.py", line 484, in __array__
    return np.asarray(self.get_duck_array(), dtype=dtype)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/common.py", line 211, in get_duck_array
    return self[key]  # type: ignore [index]
           ~~~~^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 101, in __getitem__
    return indexing.explicit_indexing_adapter(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/indexing.py", line 987, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 113, in _getitem
    original_array = self.get_array(needs_lock=False)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 92, in get_array
    ds = self.datastore._acquire(needs_lock)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 412, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 209, in _acquire_with_cache_info
    with self._optional_lock(needs_lock):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/contextlib.py", line 301, in helper
    return _GeneratorContextManager(func, args, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/contextlib.py", line 105, in __init__
    self.gen = func(*args, **kwds)
               ^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

I don't get this issue with my dca runs. I will try running hfos with quadtree now...

joe-pitt commented 1 month ago

It's actually also failing with quadtree

Traceback (most recent call last):
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/hbmcmc/run_hbmcmc.py", line 247, in <module>
    mcmc_function(**param)
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/hbmcmc/hbmcmc.py", line 442, in fixedbasisMCMC
    fp_data = basis_functions_wrapper(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/_wrapper.py", line 95, in basis_functions_wrapper
    basis_data_array = fixed_outer_regions_basis(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/_functions.py", line 193, in fixed_outer_regions_basis
    inner_region = basis_function(fp_all, start_date, emissions_name, nbasis, abs_flux, mask=mask)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/_functions.py", line 108, in quadtreebasisfunction
    quad_basis = xr.apply_ufunc(func, fps)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/computation.py", line 1270, in apply_ufunc
    return apply_dataarray_vfunc(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/computation.py", line 316, in apply_dataarray_vfunc
    result_var = func(*data_vars)
                 ^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/xarray/core/computation.py", line 825, in apply_variable_ufunc
    result_data = func(*input_data)
                  ^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/openghg_inversions/openghg_inversions/basis/algorithms/_quadtree.py", line 115, in get_quadtree_basis
    optim = scipy.optimize.dual_annealing(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/home/zh21490/.conda/envs/openghg_dev_env/lib/python3.12/site-packages/scipy/optimize/_dual_annealing.py", line 639, in dual_annealing
    raise ValueError('Bounds are not consistent min < max')
ValueError: Bounds are not consistent min < max
joe-pitt commented 1 month ago

This issue was because I was using a flux file where everything was zero. So actually the error message did lead us to find this and probably no further action is required.