Problem with _FillValue when resizing

[mortenwh]

I think we might have a problem with the fill values when resizing. It seems the fill values are also interpolated or used in the average/nearest-neighbor. We need to check further and find out how to solve it...

[asumak]

image.thumbnail(NEAREST) is possible to use instead of GDAL nearest-neighbor for data with _FillValue. Do anyone have idea for average?

[asumak]

A combination of scipy.signal.convolve2d and img.thumbnail(NEAREST) brings same results with GDAL average.

[asumak]

nearest-neighbor works fine for NaN. If some of them are NaN, they are ignored using average algorithm. If all values are NaN, they are replaced to 0. For example: resize to 50% [[nan, nan],[100 101]] --> [100.5] [[nan, nan],[nan nan] --> [0.0] Give a warning?

resize_lite() works fine for NaN values.

[asumak]

When values are NaN without "_FillValue", resize() works as the above mentioned. But with without "_FillValue" setting, NaN is replaced the value of "_FillValue".

For Example: resize to 50% (algo= -1, "_FillValue" = -10000) [[NaN, 19.69709969],[NaN, 19.69709778]] --> -4990.15136719 ( NaN is replaced to -10000 ) [[NaN, NaN],[NaN, NaN]] --> [NaN]

[asumak]

I have tested with "RS2_OK30079_PK297053_DK267373_SCWA_20120610_165905_HH_SGF".

n[2][24][8] = NaN, n[2][24][9] = NaN, n[2][25][8] = NaN, n[2][25][9] = 19.69709587, -- resize 50% --> n[2][12][4] = -7495.08 (algo = -1), 19.697 (algo = 0), -7482.63 (algo=1)

n[2][24][0] = NaN, n[2][24][1] = NaN, n[2][25][0] = NaN, n[2][25][1] = NaN, -- resize 50% --> n[2][12][0] = NaN (algo = -1), NaN (algo = 0), NaN (algo = 1)

[asumak]

BetaSigmaToIncidence() in pixelcuntions.c returns incidence angle. It is computed as:

if (beta0 != 0) incidence = asin(sigma0/beta0)*180/PI else incidence = -10000

Therefore, NaN is replaced to -10000.

[asumak]

when beta0 == 0, if BetaSigmaToIncidence() returns NAN ,

n[2][24][8] = NaN, n[2][24][9] = NaN, n[2][25][8] = NaN, n[2][25][9] = 19.69709587, -- resize 50% --> n[2][12][4] = 19.697 (algo = -1), 19.697 (algo = 0), NaN (algo=1)

n[2][24][0] = NaN, n[2][24][1] = NaN, n[2][25][0] = NaN, n[2][25][1] = NaN, -- resize 50% --> n[2][12][0] = 0.0 (algo = -1), NaN (algo = 0), NaN (algo = 1)

[asumak]

The problem comes from BetaSigmaToIncidence() in pixelfunctions because it returns -10000 if beta0 =0. Isn't it good to return NaN when beta0 = 0?

[mortenwh]

Can the pixelfunction return nan?

On 7 January 2014 16:19, Asuka Yamakawa notifications@github.com wrote:

The problem comes from BetaSigmaToIncidence() in pixelfunctions because it returns -10000 if beta0 =0. Isn't it good to return NaN when beta0 = 0?

— Reply to this email directly or view it on GitHubhttps://github.com/nansencenter/nansat/issues/42#issuecomment-31745849 .

[asumak]

Yes, the pixelfunction can return nan.

[mortenwh]

Ok, I think I let it return -10000 because I didn't find how to return nan so I guess you could try that and see if it works. I won't be able to test it myself before I'm back in the end of February though... You'll keep it in the issue branch, right?

8. jan. 2014 08:41 skrev "Asuka Yamakawa" notifications@github.com følgende:

Yes, the pixelfunction can return nan.

— Reply to this email directly or view it on GitHubhttps://github.com/nansencenter/nansat/issues/42#issuecomment-31811206 .

[asumak]

OK. RS2 problem will be solved by returning NaN in the pixelfunctions. Another problem is NetCDF. When exporting data to NetCDF, "_FillValue = -10000" is set in GDAL. The same problem occurs with the NetCDF file.

Yes, I will keep it in the branch.

[asumak]

When resizing with algorithms except for -1, NAN is returned. When exporting the resized data to NetCDF and open it, NAN values are kept as NAN. A problem is algorithm = -1 (average). I think this is same as #issue41. 2a79d3f1e9 gives a warning if data include NAN and are resized with algo=-1. Move this issue to 0.7?

[asumak]

Added tests in 0554f3a and modified the conditions to raise error in d3f745b. Morten, can you please check? if it is ok, can you merge it?

[mortenwh]

Update error messages in VRT.get_subsampled_vrt

[asumak]

Changed the error messages in bfaf672.

[mortenwh]

I have added tests, and it is still failing. The test_export function fails with a strange error for me - can you also try that?

[asumak]

ok

[asumak]

FillValue has to be replaced to np.nan for get_warped_vrt(). It was fixed in 758807c. The TestRadarsat.test_export function failed. It seems another problem. Can I make another issue for this problem?

[asumak]

Which error did you get with test_export function, Morten? Is it 'IReadBlock failed at X offset 0, Y offset 0'??

[asumak]

Regarding export function error: I got the error on my virtual machine which uses GDAL 1.11.1. My local machine and Johansen useing GDAL 1.10.1 do not show the error.

Although the virtual machine shows the error, a netCDF file is created properly. And I found a strange behavior.

n = Nansat(fileName) n.export('test.nc')

shows the error.

But

n = Nansat(fileName) print n[1] n.export('test.nc')

has no error

Which GDAL version do you use, Morten?

[mortenwh]

I use 1.10 but I won't have time to look more at this before after easter...

On 31 March 2015 at 14:08, Asuka Yamakawa notifications@github.com wrote:

Regarding export function error: I got the error on my virtual machine which uses GDAL 1.11.1. My local machine and Johansen useing GDAL 1.10.1 do not show the error.

Although the virtual machine shows the error, a netCDF file is created properly. And I found a strange behavior.

n = Nansat(fileName) n.export('test.nc') shows the error.

But

n = Nansat(fileName) print n[1] n.export('test.nc') has no error

Which GDAL version do you use, Morten?

— Reply to this email directly or view it on GitHub https://github.com/nansencenter/nansat/issues/42#issuecomment-88062703.

[mortenwh]

I have updated tests and test data. The issue is there when we export the resized complex RS2 scene (could also be there for non-complex scenes, but let's focus on this first). The incidence angles in the exported netcdf file are close to zero, which is obviously wrong.