search

worldbank / blackmarbler

Georeferenced Rasters and Statistics of Nighttime Lights from NASA Black Marble
https://worldbank.github.io/blackmarbler/
Other
16 stars 4 forks source link

`bm_raster` returns `NULL` #13

Open simon-smart88 opened 5 months ago

simon-smart88 commented 5 months ago

This has been the case for a month or so now. I've updated to the dev version but the problem persists. It seems unrelated to the sf object that I pass.

library(sf)

square_coords <- matrix(c(
  -1.1, 50.5,  
  0.9,  50.5, 
  0.9,  52.5,  
  -1.1, 52.5,  
  -1.1, 50.5   
), ncol = 2, byrow = TRUE)

square_polygon <- st_polygon(list(square_coords))
square_sf <- st_sfc(square_polygon, crs = 4326)  
sf_object <- st_sf(geometry = square_sf)

blackmarbler::bm_raster(roi_sf = sf_object,
                        product_id = "VNP46A4",
                        date = "2021",
                        bearer = Sys.getenv("NASA_bearer"),
                        quiet = FALSE)

Produces:

Processing 2 nighttime light tiles
Processing: VNP46A4.A2021001.h17v03.001.2022094115448.h5
  |========================================================================================================================| 100%
Downloading: 4.1 kB     NULL
ramarty commented 5 months ago

@simon-smart88 Thanks for flagging the issue. Was testing now, and there seem to be some issues on the NASA LAADS archive. Will keep checking to see if the problem is with the code or with the archive that the code is trying to access — and at the very least aim to include some more helpful error messages when there seems to be issues with the LAADS archive.

firatgundem commented 3 months ago

I think the problem still exists. Any improvements? Below my codes:

roi_sf <- gadm(country = "GHA", level=1, path = tempdir())

Daily data: raster for February 5, 2021

r_20210205 <- bm_raster(roi_sf = roi_sf, product_id = "VNP46A2", date = "2021-02-05", bearer = bearer) r_20210205 NULL

firatgundem commented 3 months ago

I think no one is checking this page. Is there anyone who can help? Why does bm_raster create a null data set?

ramarty commented 3 months ago

@firatgundem Sorry for the delay here! Does the dev version work for you?

install.packages("devtools") devtools::install_github("worldbank/blackmarbler")

Sometimes there's issues as the NASA LAADs archive is down which will cause NULL to be returned - I'm working on having the package return a message when this is the case, but one way to see if that's the source of the error is if you can manually download a file:

https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A4/2012/001/

Another issues could be the bearer token expired, so could be worth trying to regenerate that.

firatgundem commented 3 months ago

Image

Yes, I tried devtools as well but got the same thing. I have been trying to obtain this blackmarble dataset for more than 1 week and I could only get the null data. Could you please help me?

ramarty commented 3 months ago

Ah thanks for this!

1/ Could be worth making sure all the packages it depends on are up to date?

readr, hdf5r, dplyr, purrr, lubridate, tidyr, terra, sf, exactextractr, stringr, httr

2/ Are you on a mac or pc?

firatgundem commented 3 months ago

I use pc and my packages are up-to-date. This is the latest error that I got:

black_marble_data = bm_raster(roi_sf = shapefile, # this specifies the region of interest

Error in downloading data; bearer token likely invalid. Try regenerating the bearer token; please see this link for instructions to obtain a bearer token: https://github.com/worldbank/blackmarbler?tab=readme-ov-file#bearer-token- Processing 1 nighttime light tiles Processing: VNP46A3.A2021305.h33v12.001.2021343154743.h5 |===============================================| 100% Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1) Processing 1 nighttime light tiles Processing: VNP46A3.A2021335.h33v12.001.2022008110827.h5 |===============================================| 100% Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1)

Storing the layer names from the original raster stack

original_layer_names = names(black_marble_data)

Imputing missing values in the raster data

imputed_data = approxNA(black_marble_data, method="linear", rule=2, f=0) #addressing the NA values by linear interpolation Error: unable to find an inherited method for function ‘approxNA’ for signature ‘x = "NULL"’

It is also interesting to see these. Although it says the bearer token is likely invalid, I have just renewed it. How can it be invalid?

marcoaurelioguerrap commented 1 week ago

I'm having the same problem too, I'll try to debug the code now.

marcoaurelioguerrap commented 1 week ago

Yeah, the problem with me was the bearer key. It changed, Although I did not request the change.

simon-smart88 commented 1 week ago

@marcoaurelioguerrap

Here's a function that will stop that being an issue. It fetches an existing token or creates a new one if none exist:

get_nasa_token <- function(username, password) {

  token_url <- "https://urs.earthdata.nasa.gov/api/users/find_or_create_token"

  req <- httr2::request(token_url)

  response <- tryCatch(
    req |>
      httr2::req_auth_basic(username, password) |>
      httr2::req_method("POST") |>
      httr2::req_perform(),
    httr2_http_401 = function(cnd){NULL}
  )

  if (httr2::resp_status(response) == 200) {
    body <- response %>% httr2::resp_body_json()
    token <- body$access_token
    return(token)
  } else {
    return()
  }

}
ramarty commented 1 week ago

@simon-smart88 Thanks for writing that function. Would love to add to the package + note in the documentation if OK with you.

If you're willing to do a pull request that adds the function in here that'd be great - or I'm happy to just add myself if you prefer. Then I can test as well, add that to the documentation, then send the update to CRAN.

(I may also add an option in the bm_raster/bm_extract functions for a user to either include (1) bearer or (2) username/password).

simon-smart88 commented 1 week ago

Yes sure I will do - I think it probably needs a few tweaks to be robust to all eventualities though so will work on that and submit a PR.

ramarty commented 1 week ago

Amazing, thanks @simon-smart88!