search

pytroll / pyresample

Geospatial image resampling in Python
http://pyresample.readthedocs.org
GNU Lesser General Public License v3.0
346 stars 94 forks source link

Pyresample/geometry.py resampling error related to dask. #425

Closed zhaoyutim closed 2 years ago

zhaoyutim commented 2 years ago

Code Sample, a minimal, complete, and verifiable piece of code

# Your code here
# lon lat from VNP03IMG
lon = scn['i_lon'].values
lat = scn['i_lat'].values
# create area definition and do resample
area = create_area_def(area_id="area", projection='WGS84', shape=(lat.shape[1], lat.shape[0]), lon=lon, lat=lat)
new_scn = scn.resample(destination=area)

Problem description

Hi Team,

I was trying to resample a VIIRS image from VNPIMG02 and VNPIMG03, however I encounter this issue when I was doing the resample. It got an error related to dask like : "ValueError: zero-size array to reduction operation fmax which has no identity" I located the problem and provided a quick fix as below, it seems related to dask computation for nanmax. The example array I was using is extracted from longitude of my satellite image.

Thanks in advance! Yu Zhao

Expected Output

Quick fix starts here

xarr = np.load('example.npy') print(xarr)

my implementation

xarr = da.from_array(xarr) xmin = np.nanmin(xarr[xarr >= 0].compute()) xmax = np.nanmax(xarr[xarr < 0].compute()) + 360 print(xmin, xmax)

implementation in the package (this implementation would report error)

xmin2 = np.nanmin(xarr[xarr >= 0]) xmax2 = np.nanmax(xarr[xarr < 0]) + 360 xmin2, xmax2 = da.compute(xmin2, xmax2) print(xmin2, xmax2)

Actual Result, Traceback if applicable

Versions of Python, package at hand and relevant dependencies

satpy 0.27 pyresample 1.19

djhoese commented 2 years ago

Could you provide:

  1. The full traceback of the original ValueError you got? This will show us where in the code the problem is and how it was called.
  2. The location of where your changes would show up in pyresample (a github link to the exact line of code please)? I don't know where the xmin/xmax you are referring to are created or used.
zhaoyutim commented 2 years ago

Hi thanks for your response, here are what you need!

  1. C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\pyproj\crs\crs.py:1256: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems return self._crs.to_proj4(version=version) Traceback (most recent call last): File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\satpy\scene.py", line 722, in _resampled_scene destination_area = destination_area.freeze(finest_area) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\pyresample\geometry.py", line 976, in freeze corners = self._compute_bound_centers(proj_dict, lonslats) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\pyresample\geometry.py", line 985, in _compute_bound_centers return self._compute_bound_centers_dask(proj_dict, lons, lats) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\pyresample\geometry.py", line 1033, in _compute_bound_centers_dask xmin, xmax = da.compute(xmin, xmax) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\base.py", line 567, in compute results = schedule(dsk, keys, kwargs) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\threaded.py", line 87, in get kwargs File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\local.py", line 514, in get_async raise_exception(exc, tb) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\local.py", line 325, in reraise raise exc File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\local.py", line 223, in execute_task result = _execute_task(task, data) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\core.py", line 121, in _execute_task return func((_execute_task(a, cache) for a in args)) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\optimization.py", line 963, in call return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args))) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\core.py", line 151, in get result = _execute_task(task, cache) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\core.py", line 121, in _execute_task return func((_execute_task(a, cache) for a in args)) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\dask\utils.py", line 34, in apply return func(*args, kwargs) File "<__array_function__ internals>", line 6, in nanmax File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\numpy\lib\nanfunctions.py", line 434, in nanmax res = np.fmax.reduce(a, axis=axis, out=out, kwargs) ValueError: zero-size array to reduction operation fmax which has no identity

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "D:/LowResSatellitesService/dev.py", line 26, in pipeline.processing(date, roi) File "D:\LowResSatellitesService\ProcessingPipeline\processing_pipeline.py", line 141, in processing self.read_and_projection(date, roi, dir_data) File "D:\LowResSatellitesService\ProcessingPipeline\processing_pipeline.py", line 56, in read_and_projection new_scn = scn.resample(destination=area) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\satpy\scene.py", line 822, in resample reduce_data=reduce_data, **resample_kwargs) File "C:\Users\Yu\PycharmProjects\LowResSatellitesService\venv\lib\site-packages\satpy\scene.py", line 724, in _resampled_scene raise ValueError("No dataset areas available to freeze " ValueError: No dataset areas available to freeze DynamicAreaDefinition.

2. https://github.com/pytroll/pyresample/blob/29d37a4089c9bd5d56c05ce715948a5f7e8cebcc/pyresample/geometry.py#L1077

zhaoyutim commented 2 years ago

Sorry it should be this line, but I found the version I was using is different from what you have now. I think it has been fixed in current version. Thanks for checking it! https://github.com/pytroll/pyresample/blob/29d37a4089c9bd5d56c05ce715948a5f7e8cebcc/pyresample/geometry.py#L1110

djhoese commented 2 years ago

Ah ok, thanks @zhaoyutim. This work was done as part of #346. If you could upgrade your version of pyresample and let me know if it fixes your issue that'd be great. I'm going to assume it does so I'll close this. Thanks for reporting the problem either way.