Problem with NA values for LST downscaling

[ghost]

I have encountered problems related to downscal errors.zip

ing time series raster stack from 2002 to 2017 Terra MODIS LST data using two predictors -- DEM and Terra MODIS NDVI. I have created raster stack for predictors and raster layer for Modis LST. My study area is South Alps that's why I use DEM as additional explanatory variable. As an attachment I am sending my code with datasets and short explanation as well. I would be really grateful for help. Thank you!

[pierreroudier]

@aniluap5 unfortunately the data you saved (using the save function) refer to local files on your computer.

Best sending them (ideally subsets) as GeoTIFF.

[ghost]

Hi Pierre,

Sorry for my omission. Below you will find geotiffs as input layers.

errors_GEOtiffs.zip

[pierreroudier]

@aniluap5 dissever version 0.2-3 (on its way to CRAN right now) should hopefully fix the issue. Please test!

Thanks for the bug report.

[ghost]

Hi Pierre,

Unfortunately, the problem is not solved. Previously, I have used this version. Now, I tried again using this method. As a result I got the same errors which you can see in an attachment. NA values for rasters remain problematic issue anyway.

[pierreroudier]

@aniluap5 Thanks, can you just confirm the version of dissever you are running (the output of packageVersion("dissever")?

Also, could you post here the part of your script where the error occurs?

Cheers

[pierreroudier]

@aniluap5 Note that there seem to be a lot of NAs/missing values in your predictors.

I ended up with the following result:

The pixels that remain in the prediction are the only pixels that don't have missing values throughout your time series.

I'm guessing that means you'd need to do some gap-filling prior to dissever.

[pierreroudier]

@aniluap5 Following my last post -- note that I have a quick get around using an average operator applied on groups of 5 LST images:

ndvi <- stack("Terra_bigNDVI.tif")
dem <- stack("big.dem_250m.tif")
dem <- projectRaster(dem[[1]], ndvi, method = 'bilinear')

# Cutting the NDVI indices in groups of 5
idx <- cut(1:225, breaks = seq(0, 225, by = 5))

# Creating a look up table referencing every NDVI map to 
# its group
lut <- data.frame(i  = 1:225, idx = as.numeric(idx))

# Splitting the stack of NDVI layers into a list of goups of 5
foo <- plyr::dlply(lut, "idx", function(x) ndvi[[x$i]])

# For each group, compute the average NDVI over 5 days
bar <- lapply(foo, function(x) calc(x, mean, na.rm = TRUE))

# Convert the result to a stack
baz <- stack(bar)

# Stack the 5 days averages of NDVI with DEM
res_rf <- dissever(coarse = lst, fine = stack(dem, baz), method = "rf", p = p_train, min_iter = min_iter, max_iter = max_iter)

# Plot result
plot(res_rf)

[ghost]

Dear Pierre,

I have checked my last version of R and it is ‘0.2.3’. The errors occurs in the following part:

spatial downscaling using RANDOM FOREST regression - using whole imagery

(the first one from time-series stack)

it does not work because of Na values

min_iter <- 5 max_iter<-15 p_train <- 0.025

res_rfwhole <- dissever(

• coarse = aggregate(Terra_LST_final[[1]],fun= mean,fact=1, na.rm=TRUE, inf.rm=TRUE), *
• fine = aggregate(Terrastack_NDVI_DEM_250m[[1]],fun= mean,fact=1,na.rm=TRUE, inf.rm=TRUE), #na.rm and inf.rm do not work :(*
• method = "rf", # regression method used for disseveration*
• p = p_train, # proportion of pixels sampled for training regression model*
• min_iter = min_iter, # minimum iterations*
• max_iter = max_iter # maximum iterations*

)

• In this code I take into account bigger subset. As a result I got the error something like this: *

I use images direct from list of rasters (e.g. Terra_LST_final[[1]] and Terrastack_NDVI_DEM_250m[[1]]). I think that script has problems with many Na values.You are right it is better to use gap-filling procedure before downscaling.

I think I can't cut my raster stack grouping it into 5-band stacks. In my opinion there is too big difference in time between my 'bands' (16-day revisit time ). And of course, vegetation changes in time. Maybe it would be better to use gap-filling option for each band separately. What do you thin about it?

I would be grateful for your opinion!

Best, Paulina

On Mon, Apr 23, 2018 at 6:24 AM, Pierre Roudier notifications@github.com wrote:

@aniluap5 https://github.com/aniluap5 Following my last post -- note that I have a quick get around using an average operator applied on groups of 5 LST images:

ndvi <- stack("Terra_bigNDVI.tif") dem <- stack("big.dem_250m.tif") dem <- projectRaster(dem[[1]], ndvi, method = 'bilinear')

Cutting the NDVI indices in groups of 5

idx <- cut(1:225, breaks = seq(0, 225, by = 5))

Creating a look up table referencing every NDVI map to

its group

lut <- data.frame(i = 1:225, idx = as.numeric(idx))

Splitting the stack of NDVI layers into a list of goups of 5

foo <- plyr::dlply(lut, "idx", function(x) ndvi[[x$i]])

For each group, compute the average NDVI over 5 days

bar <- lapply(foo, function(x) calc(x, mean, na.rm = TRUE))

Convert the result to a stack

baz <- stack(bar)

Stack the 5 days averages of NDVI with DEM

res_rf <- dissever(coarse = lst, fine = stack(dem, baz), method = "rf", p = p_train, min_iter = min_iter, max_iter = max_iter)

Plot result

plot(res_rf)

[image: rplot01] https://user-images.githubusercontent.com/515259/39106923-d5faf28a-4712-11e8-9c78-c77f7d679ca5.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pierreroudier/dissever/issues/6#issuecomment-383450580, or mute the thread https://github.com/notifications/unsubscribe-auth/AjwyY7piFRPHERvlbB5xxTCM-_N_VLtvks5trVeTgaJpZM4TMams .

[pierreroudier]

@aniluap5

From this piece of code:

res_rfwhole <- dissever(
  coarse = aggregate(Terra_LST_final[[1]],fun= mean,fact=1, na.rm=TRUE, inf.rm=TRUE),
  fine = aggregate(Terrastack_NDVI_DEM_250m[[1]],fun= mean,fact=1,na.rm=TRUE, inf.rm=TRUE),
  method = "rf", 
  p = p_train, 
  min_iter = min_iter, 
  max_iter = max_iter
)

Could you please run the following:

coarse <- aggregate(Terra_LST_final[[1]],fun= mean,fact=1, na.rm=TRUE, inf.rm=TRUE)
fine <- aggregate(Terrastack_NDVI_DEM_250m[[1]],fun= mean,fact=1,na.rm=TRUE, inf.rm=TRUE)
writeRaster(coarse, "coarse.tif")
writeRaster(fine, "fine.tif")

and post the resulting GeoTIFF (coarse.tif and fine.tif) here. I struggle to reproduce your bug.

Also:

I think that script has problems with many Na values.You are right it is better to use gap-filling procedure before downscaling. I think I can't cut my raster stack grouping it into 5-band stacks. In my opinion there is too big difference in time between my 'bands' (16-day revisit time ). And of course, vegetation changes in time. Maybe it would be better to use gap-filling option for each band separately. What do you thin about it?

Is the 16-days revisit time for NDVI? If so, it must be a very cloudy region to still display so many NAs! Anyway, moving forward, I think I would try and apply gap-filling to that NDVI stack. There are many ways to do this:

• On individual rasters, independently, this is basically just spatial interpolation
• On the raster time serie, you can interpolate in both time and space

For the former, people have used all sorts of methods such as DINEOF. For the latter, GRASS GIS is a great tool if you master it (see the r.series.lwr command).