[icoscp-core] NumPy > 2.0 incompatibility

[olivierbonte]

Hello,

When running the example code given here, there seems to be a problem with the code for Numpy 2.0 and newer. To solve this issue I think you could

1. Restrict to numpy<2.0 in https://github.com/ICOS-Carbon-Portal/data/blob/master/src/main/python/icoscp_core/pyproject.toml#L20
2. Update to the NumPy 2.0 convention as suggested under "message" in the error below (seems recommended).

System information:

• OS: Windows10
• Python installed as conda env create -f environment.yml with the environment.yml defined below:

  name: icos_error_fix_env
  channels:
  - conda-forge
  dependencies:
  - python=3.10
  - matplotlib
  - jupyterlab
  - pip
  - pip:
  - icoscp

  Specifically I got following error:

  
  "name": "AttributeError",
  "message": "`newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.",
  "stack": "---------------------------------------------------------------------------
  AttributeError                            Traceback (most recent call last)
  Cell In[1], line 26
   23 merged_co2 = pd.DataFrame(columns=['TIMESTAMP', 'co2'])
   25 # batch-fetch the interesting columns and iterate through the results
  ---> 26 for dobj, arrs in data.batch_get_columns_as_arrays(se_co2_from_100, ['TIMESTAMP', 'co2']):
   27     st_uri = dobj.station_uri
   28     # ICOS atmospheric station URIs end with underscore followed by a 3-letter station ID
   29     # this ID is convenient to use as a suffix to rename 'co2' with

File c:\Users\obonte\AppData\Local\miniconda3\envs\icos_error_fix_env\lib\site-packages\icoscp_core\dataclient.py:228, in DataClient.batch_get_columns_as_arrays(self, dobjs, columns, keep_bad_data) 225 \treport_cpb_file_read(self._conf, hashes=hashes, columns=columns) 227 for dobj, codec in dobj_codecs: --> 228 \tyield dobj, self._get_columns_as_arrays(codec, keep_bad_data)

File c:\Users\obonte\AppData\Local\miniconda3\envs\icos_error_fix_env\lib\site-packages\icoscp_core\dataclient.py:241, in DataClient._get_columns_as_arrays(self, codec, keep_bad_data) 239 url = self._conf.data_service_base_url + '/cpb' 240 resp = http_request(url, \"Fetching binary from \" + url, \"POST\", headers, codec.json_payload) --> 241 return codec.parse_cpb_response(resp.fp, keep_bad_data)

File c:\Users\obonte\AppData\Local\miniconda3\envs\icos_error_fix_env\lib\site-packages\icoscp_core\cpb.py:191, in Codec.parse_cpb_response(self, resp, keep_bad_data) 190 def parse_cpb_response(self, resp: io.BufferedReader, keep_bad_data: bool) -> ArraysDict: --> 191 \treturn self._parse_from_buff(resp, None, keep_bad_data)

File c:\Users\obonte\AppData\Local\miniconda3\envs\icos_error_fix_env\lib\site-packages\icoscp_core\cpb.py:225, in Codec._parse_from_buff(self, buff, col_offsets, keep_bad_data) 221 \tarr = _type_post_process(arr, col.value_format_uri) 223 \tif sys.byteorder == 'little': 224 \t\t# cpb files are all big-endian, fixing for Pandas compatibility on little endian systems: --> 225 \t\tarr = arr.byteswap().newbyteorder() 226 \tres[col.label] = arr 228 if not keep_bad_data and self._ci.good_flags:

AttributeError: newbyteorder was removed from the ndarray class in NumPy 2.0. Use arr.view(arr.dtype.newbyteorder(order)) instead."

I tried to make a pull request myself, but I did not manage to set up the icoscp_core package in development mode (I tried using flit). Therefore, a bit more information on how to contribute (e.g. under https://github.com/ICOS-Carbon-Portal/data/tree/master/src/main/python/icoscp_core#getting-started) would be nice.

Thank you in advance,
Kind regards
Olivier Bonte 

[mirzov]

Hi Olivier,

Thanks for the fantastic bug report! I will opt for solution 1 for now, as our higher-level lib depends on pandas, and that one depends on numpy 1.x (and probably will take a while to upgrade to 2.x).

[olivierbonte]

No worries, thanks for the quick fix!