Wrong datetime when opening file with numpy 2.0.0

renaudjester commented 3 months ago

What happened?

I am trying to open netcdf files or even zarr files but with the latest version of numpy==2.0.0, the values for the time dimensions are wrong (by some seconds). It happens when I open directly netcdf files but also some zarr files.

What did you expect to happen?

I am expecting the datetime for the time dimension to work as it was with numpy<2.0.0

Minimal Complete Verifiable Example

# download any of the files here: https://data.marine.copernicus.eu/product/IBI_MULTIYEAR_PHY_005_002/files?subdataset=cmems_mod_ibi_phy_my_0.083deg-3D_P1M-m_202012&path=IBI_MULTIYEAR_PHY_005_002%2Fcmems_mod_ibi_phy_my_0.083deg-3D_P1M-m_202012%2F2021%2F

import xarray

ds = xr.open_dataset(
    "CMEMS_v5r1_IBI_PHY_MY_PdE_01mav_20211001_20211031_R20230101_RE01.nc"
)

print(ds)
# with numpy<2.0.0: (time) datetime64[ns] 2021-10-16T12:00:00
# with numpy==2.0.0: (time) datetime64[ns] 8B 2021-10-16T11:58:44.279541760

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[ ] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.
[X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

<xarray.Dataset> Size: 85MB
Dimensions:    (time: 1, longitude: 289, latitude: 361, depth: 50)
Coordinates:
  * time       (time) datetime64[ns] 8B 2021-10-16T11:58:44.279541760
  * longitude  (longitude) float32 1kB -19.0 -18.92 -18.83 ... 4.833 4.917 5.0
  * latitude   (latitude) float32 1kB 26.0 26.08 26.17 ... 55.83 55.92 56.0
  * depth      (depth) float32 200B 0.5058 1.556 2.668 ... 5.292e+03 5.698e+03
Data variables:
    thetao     (time, depth, latitude, longitude) float32 21MB ...
    so         (time, depth, latitude, longitude) float32 21MB ...
    uo         (time, depth, latitude, longitude) float32 21MB ...
    vo         (time, depth, latitude, longitude) float32 21MB ...
    zos        (time, latitude, longitude) float32 417kB ...
    bottomT    (time, latitude, longitude) float32 417kB ...
    mlotst     (time, latitude, longitude) float32 417kB ...

Anything else we need to know?

I tried to do something like this:

import numpy as np
import pandas as pd
import xarray as xr

np.random.seed(0)
temperature = 15 + 8 * np.random.randn(2, 3, 4)
precipitation = 10 * np.random.rand(2, 3, 4)
lon = [-99.83, -99.32]
lat = [42.25, 42.21]
instruments = ["manufac1", "manufac2", "manufac3"]
time = pd.date_range("2014-09-06", periods=4)
reference_time = pd.Timestamp("2014-09-05")

ds = xr.Dataset(
    data_vars=dict(
        temperature=(["loc", "instrument", "time"], temperature),
        precipitation=(["loc", "instrument", "time"], precipitation),
    ),
    coords=dict(
        lon=("loc", lon),
        lat=("loc", lat),
        instrument=instruments,
        time=time,
        reference_time=reference_time,
    ),
    attrs=dict(description="Weather related data."),
)
print(ds)

# save to netcdf
ds.to_netcdf("lololol.nc")

# open the netcdf file
ds = xr.open_dataset("lololol.nc")

# print the dataset
print(ds)

But it seemed to work as expected so maybe the example is too simple. However, anything else I tried had the error.

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.11.6 (main, May 17 2024, 17:39:14) [Clang 15.0.0 (clang-1500.3.9.4)] python-bits: 64 OS: Darwin OS-release: 23.5.0 machine: arm64 processor: arm byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: ('en_GB', 'UTF-8') libhdf5: 1.14.3 libnetcdf: 4.9.2 xarray: 2024.6.0 pandas: 2.2.2 numpy: 2.0.0 scipy: None netCDF4: 1.7.1.post1 pydap: None h5netcdf: None h5py: None zarr: None cftime: 1.6.4 nc_time_axis: None iris: None bottleneck: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 69.5.1 pip: 24.0 conda: None pytest: None mypy: None IPython: None sphinx: None

keewis commented 3 months ago

I think this is a duplicate of #9179, which was fixed in #9182. If possible, can you try installing from main to see if that works? Otherwise the next release should be in a couple of weeks.

renaudjester commented 3 months ago

Thanks for the fast answer @keewis ! Indeed, it seems to be fixed when I am trying the same example from main :D

Sorry, I haven't seen the other issue, I think this one can be closed.

keewis commented 3 months ago

No worries, and thanks for taking the time to write up the report.

pydata / xarray