TUW-GEO / repurpose

Routines for the conversion of image formats to time series and vice versa.
Other
4 stars 4 forks source link

ValueError: assignment destination is read-only with parallel processing #28

Open pstradio opened 1 month ago

pstradio commented 1 month ago

With version v0.11 I am getting the following when trying to ts2img and > 1 n_proc.

ERROR:root:The following ERROR was raised in the parallelized function `_convert` but was ignored due to the chosen settings: Traceback (most recent call last):
  File "/home/pstradio/miniforge3/envs/rep/lib/python3.9/site-packages/repurpose/process.py", line 175, in run_with_error_handling
    r = FUNC(**kwargs)
  File "/home/pstradio/miniforge3/envs/rep/lib/python3.9/site-packages/repurpose/ts2img.py", line 59, in _convert
    writer.write_ts(ts, gpi)
  File "/home/pstradio/miniforge3/envs/rep/lib/python3.9/site-packages/repurpose/stack.py", line 357, in write_ts
    self.ds[var].values[t, row, col] = df[var].values
ValueError: assignment destination is read-only

I'm running the following:

...

inreader = CustomeReader()  # just a wrapper around a pynetcf reader
outgrid = BasicGrid(lon2d.flatten(), lat2d.flatten(), shape=(lats.size, lons.size))
dates = pd.date_range("2015-04-01T06:00:00", "2022-09-29T18:00:00", freq="12h")

conv = Ts2Img(inreader, outgrid, dates,)

conv.calc(
    path_out=os.path.join(BASEPATH, 'images'),
    format_out='slice',
    var_attrs=dict(sm=dict(long_name='soil moisture', units='m3 m-3')),
    n_proc=6,
    img_buffer=190,
    zlib=True,
)

I can reproduce the issue on 2 different machines.

wpreimes commented 1 month ago

hmm it looks like writing to the xarray Dataset (which should be in memory) is not possible. Could it be that the outfile already exists and you cannot overwrite it? Otherwise, I have noticed that some tests failed when n_proc>1. I have fixed this now, maybe it was related. Can you try again with the master branch after #29 is merged (but I think your problem is a different one...).

pstradio commented 1 month ago

I tried changing the parallelization backend and it works with threading - I will include in a PR to be able to specify the be