`bm_raster` returns `NULL`

[simon-smart88]

This has been the case for a month or so now. I've updated to the dev version but the problem persists. It seems unrelated to the sf object that I pass.

library(sf)

square_coords <- matrix(c(
  -1.1, 50.5,  
  0.9,  50.5, 
  0.9,  52.5,  
  -1.1, 52.5,  
  -1.1, 50.5   
), ncol = 2, byrow = TRUE)

square_polygon <- st_polygon(list(square_coords))
square_sf <- st_sfc(square_polygon, crs = 4326)  
sf_object <- st_sf(geometry = square_sf)

blackmarbler::bm_raster(roi_sf = sf_object,
                        product_id = "VNP46A4",
                        date = "2021",
                        bearer = Sys.getenv("NASA_bearer"),
                        quiet = FALSE)

Produces:

Processing 2 nighttime light tiles
Processing: VNP46A4.A2021001.h17v03.001.2022094115448.h5
  |========================================================================================================================| 100%
Downloading: 4.1 kB     NULL

[ramarty]

@simon-smart88 Thanks for flagging the issue. Was testing now, and there seem to be some issues on the NASA LAADS archive. Will keep checking to see if the problem is with the code or with the archive that the code is trying to access — and at the very least aim to include some more helpful error messages when there seems to be issues with the LAADS archive.

[firatgundem]

I think the problem still exists. Any improvements? Below my codes:

roi_sf <- gadm(country = "GHA", level=1, path = tempdir())

Daily data: raster for February 5, 2021

r_20210205 <- bm_raster(roi_sf = roi_sf, product_id = "VNP46A2", date = "2021-02-05", bearer = bearer) r_20210205 NULL

[firatgundem]

I think no one is checking this page. Is there anyone who can help? Why does bm_raster create a null data set?

[ramarty]

@firatgundem Sorry for the delay here! Does the dev version work for you?

install.packages("devtools") devtools::install_github("worldbank/blackmarbler")

Sometimes there's issues as the NASA LAADs archive is down which will cause NULL to be returned - I'm working on having the package return a message when this is the case, but one way to see if that's the source of the error is if you can manually download a file:

https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A4/2012/001/

Another issues could be the bearer token expired, so could be worth trying to regenerate that.

[firatgundem]

Yes, I tried devtools as well but got the same thing. I have been trying to obtain this blackmarble dataset for more than 1 week and I could only get the null data. Could you please help me?

[ramarty]

Ah thanks for this!

1/ Could be worth making sure all the packages it depends on are up to date?

readr, hdf5r, dplyr, purrr, lubridate, tidyr, terra, sf, exactextractr, stringr, httr

2/ Are you on a mac or pc?

[firatgundem]

I use pc and my packages are up-to-date. This is the latest error that I got:

black_marble_data = bm_raster(roi_sf = shapefile, # this specifies the region of interest

• product_id = "VNP46A3", # this specifies the monthly luminosity data
• date = seq.Date(from = ymd("2021-10-01"), to = ymd("2021-12-01"), by = "month"), # this specifies the date range
• bearer = bearer, # this is the API bearer for NASA LAADS DAAC from Step 3
• variable = c("AllAngle_Composite_Snow_Free"), # this selects the layer we're interested in i.e. satellite angle and snow-free observations
• quality_flag_rm = c(255, 2)) # this drops bad quality observations and coverts them to NA values) Processing 1 nighttime light tiles Processing: VNP46A3.A2021274.h33v12.001.2021321133308.h5 |===============================================| 100% Error in downloading data No encoding supplied: defaulting to UTF-8.

  504 Gateway Time-out

  ---

  nginx

Error in downloading data; bearer token likely invalid. Try regenerating the bearer token; please see this link for instructions to obtain a bearer token: https://github.com/worldbank/blackmarbler?tab=readme-ov-file#bearer-token- Processing 1 nighttime light tiles Processing: VNP46A3.A2021305.h33v12.001.2021343154743.h5 |===============================================| 100% Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1) Processing 1 nighttime light tiles Processing: VNP46A3.A2021335.h33v12.001.2022008110827.h5 |===============================================| 100% Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1)

Storing the layer names from the original raster stack

original_layer_names = names(black_marble_data)

Imputing missing values in the raster data

imputed_data = approxNA(black_marble_data, method="linear", rule=2, f=0) #addressing the NA values by linear interpolation Error: unable to find an inherited method for function ‘approxNA’ for signature ‘x = "NULL"’

It is also interesting to see these. Although it says the bearer token is likely invalid, I have just renewed it. How can it be invalid?