Retrieving measurements fails with KeyError: 'Code'

[jaapschellekens]

• Data Distributie Laag. Service from Rijkswaterstaat for distributing water quantity data. version:
• Python version:
• Operating System:

Description

Both in our test and the (commented out) units tests in test_ddlpy: 'test_measuerments' and 'test_measusrements_long' the retrieval fails and so do the tests. I expected that the measurements would be retrieved.

We run ddlpy daily and have found that the failures started around 2021-10-26. It may be a small change in the RWS api that causes this

What I Did

After uncommenting the two tests in test_ddlpy i run:

python setup.py test

        start_date = datetime.datetime(1951, 11, 1)
        end_date = datetime.datetime(1953, 4, 1)
>       measurements = ddlpy.measurements(
            location, start_date=start_date, end_date=end_date
        )

tests/test_ddlpy.py:42:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ddlpy/ddlpy.py:163: in measurements
    measurement = _measurements_slice(location, start_date=start_date_i, end_date=end_date_i)
ddlpy/ddlpy.py:96: in _measurements_slice
    'Code': location['Code']
/opt/miniconda3/lib/python3.8/site-packages/pandas/core/series.py:882: in __getitem__
    return self._get_value(key)
/opt/miniconda3/lib/python3.8/site-packages/pandas/core/series.py:990: in _get_value
    loc = self.index.get_loc(label)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Index(['Locatie_MessageID', 'Coordinatenstelsel', 'X', 'Y', 'Naam',
       'Parameter_Wat_Omschrijving', 'Compartiment...id.Code',
       'Hoedanigheid.Omschrijving', 'Parameter.Code',
       'Parameter.Omschrijving'],
      dtype='object')
key = 'Code', method = None, tolerance = None

    def get_loc(self, key, method=None, tolerance=None):
        """
        Get integer location, slice or boolean mask for requested label.

        Parameters
        ----------
        key : label
        method : {None, 'pad'/'ffill', 'backfill'/'bfill', 'nearest'}, optional
            * default: exact matches only.
            * pad / ffill: find the PREVIOUS index value if no exact match.
            * backfill / bfill: use NEXT index value if no exact match
            * nearest: use the NEAREST index value if no exact match. Tied
              distances are broken by preferring the larger index value.
        tolerance : int or float, optional
            Maximum distance from index value for inexact matches. The value of
            the index at the matching location most satisfy the equation
            ``abs(index[loc] - key) <= tolerance``.

        Returns
        -------
        loc : int if unique index, slice if monotonic index, else mask

        Examples
        --------
        >>> unique_index = pd.Index(list('abc'))
        >>> unique_index.get_loc('b')
        1

        >>> monotonic_index = pd.Index(list('abbc'))
        >>> monotonic_index.get_loc('b')
        slice(1, 3, None)

        >>> non_monotonic_index = pd.Index(list('abcb'))
        >>> non_monotonic_index.get_loc('b')
        array([False,  True, False,  True])
        """
        if method is None:
            if tolerance is not None:
                raise ValueError(
                    "tolerance argument only valid if using pad, "
                    "backfill or nearest lookups"
                )
            casted_key = self._maybe_cast_indexer(key)
            try:
                return self._engine.get_loc(casted_key)
            except KeyError as err:
>               raise KeyError(key) from err
E               KeyError: 'Code'

/opt/miniconda3/lib/python3.8/site-packages/pandas/core/indexes/base.py:2900: KeyError
------------------------------------------------------------------------ Captured stderr call -------------------------------------------------------------------------
  0%|          | 0/17 [00:00<?, ?it/s]
========================================================================== warnings summary ===========================================================================
.eggs/pytest-6.2.3-py3.8.egg/_pytest/config/__init__.py:1233
  /Users/jschellekens/src/ddlpy/.eggs/pytest-6.2.3-py3.8.egg/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: collect_ignore

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================================================= short test summary info =======================================================================
FAILED tests/test_ddlpy.py::test_measurements - KeyError: 'Code'
FAILED tests/test_ddlpy.py::test_measurements_long - KeyError: 'Code'
=============================================================== 2 failed, 8 passed, 1 warning in 26.47s ===============================================================
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-n0u4az2n'

[ThiesVanLoon]

solution #11

[SiggyF]

Thanks for your bug report, this should be fixed now.

[veenstrajelmer]

Another release of ddlpy was created today and all docs and examples were updated. Please open a new issue if you encounter any bugs.