Open Eisbrenner opened 8 months ago
This is weird, I tried to run the package tests (incidentally found a different issue which is fixed in the master) and I couldn't reproduce your error. Then I tried using a different test and file and the first time I run it I could reproduce your error, however after that running the same test, same data, the code completes without errors. I did this a few times, including using the latest release so as to remove the latest change I did to fix the other unrelated issue. Again no errors the code still runs fine. So I'm not sure why this is happening I will keep on testing see if I can again reproduce your error again.
This is a shot in the dark but could you try to update xarray to 2024.2.0, it was only released a couple of days ago, we noticed a similar issue with reindex in 2024.1.1 (running the xarray tutorial) which has now disappeared. I think in between when I reproduced your error the first and only time and rerun the test the environment I was using got updated.
Yes, I'll try that tomorrow!
I checked now with updated versions, but I still get the error sometimes (more specifically in 59/360 slices).
Versions:
xarray 2024.2.0 N-D labeled arrays and datasets in Python
xmhw 0.9.4.dev6 87722bc 'Marine heatwave detection code using xarray'
Not sure what to say, as I couldn't get it anymore also I used exactly the same data when I briefly experienced it and in all the other attempts. I will try again, but unfortunately I don't have any work time allocated to work on this anymore
I will see with those that failed if any of them fail consistently and report back in a few days. Maybe I find something reproducible.
By iteratively re-running failed calculations, I was able to get all slices done except one. But that final slice throws a different error. So, what ever the issue is, there is no way for me to reproduce it reliably.
For me this is then resolved, tho I guess for documentation purposes you may want to keep this issue open?
And, regarding the remaining final error message, I am not sure if that error is related to the original error at all, but I'll share it anyway :)
The error occurs for longitude grid-points 1136 to 1140 (time: 1981-09-01 to 2023-02-16, lat: all).
/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/xmhw/identify.py:314: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
gaps_shifted = gaps_shifted.fillna(value=True)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/calculate_mhws.py", line 112, in <module>
calculate_mhws()
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/xmhw/identify.py:314: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
gaps_shifted = gaps_shifted.fillna(value=True)
return ctx.invoke(self.callback, **ctx.params)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/calculate_mhws.py", line 108, in calculate_mhws
DetectMHWs().update(n_chunks, n_iter, slc).calculate()
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/calculate_mhws.py", line 83, in calculate
mhw_chunk, intermediate_chunk = detect(
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/xmhw/xmhw.py", line 454, in detect
results = dask.compute(mhwls)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/dask/base.py", line 663, in compute
results = schedule(dsk, keys, **kwargs)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/xmhw/identify.py", line 399, in define_events
dfmhw = mhw_features(df, len(idxarr) - 1, tdim, dims)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/xmhw/features.py", line 89, in mhw_features
df = agg_df(dftime, tdim, dims)
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/xmhw/features.py", line 157, in agg_df
dfout.loc[:,d] = val[0]
File "/proj/bolinc/users/x_ezrei/Notebooks/mhws/.venv/lib/python3.10/site-packages/pandas/core/indexing.py", line 1848, in _setitem_with_indexer
raise ValueError(
ValueError: cannot set a frame with no defined index and a scalar
Hi there,
I don't understand the error message down below, could someone here give me a hint what it is that I am doing wrong?
The code just takes 2-grid-points-longitude wide strips of the globe and calculates the climatology and MHWs (I intend to use wider strips, but for now, 2 is the smallest possible width. I run into some internal issue with multi-indexing with 1 grid-point width).
I like to run this for the whole globe, but with the speeds I am seeing I probably need to split it up. Ultimately, sending it to multiple nodes on a super computer to get the total time below a week. If I extrapolate, it looks like I can do ~45 grid points longitude in 12 hours on a single node.
The data is the OISSTv2 dataset, so nothing out of the ordinary.
versions
code
error