"Failed to download" error for ALOS FNF Microsoft PC dataset.

[laurenkwick]

Hello!

Everything with the package is amazing so far. I am having some trouble with accessing the ALOS FNF dataset using the get_stac_data function. I've spent some time debugging this, but am currently feeling stuck.

To access ALOS FNF data from the planetary computer, I am running this code:

alos_coll <- "alos-fnf-mosaic"
alos_assets <- "C"
pc_stac_source <- "https://planetarycomputer.microsoft.com/api/stac/v1/"

rsi::get_stac_data(
  aoi = sf_poly,
  start_date = "2015-01-01",
  end_date = "2021-12-31",
  asset_names = alos_assets,
  pixel_x_size = 25,
  pixel_y_size = 25,
  stac_source = pc_stac_source,
  collection = alos_coll,
  sign_function = rsi::sign_planetary_computer,
  download_function = rsi::rsi_download_rasters,
  composite_function = NULL,
  output_filename = tempfile(pattern="fnf_df_", tmpdir=tempdir(), fileext=".tif")
)

After attempting to run, I am seeing these errors:

Error: [rast] file does not exist: /var/folders/2s/hq96vnhj3w37048l_1m9r2j00000gq/T//Rtmpc4xFEv/filede08ae0aa18.tif
In addition: Warning messages:
1: Failed to download N11W085_16_FNF from 2016-01-01T00:00:00Z 
2: Failed to download N11W084_16_FNF from 2016-01-01T00:00:00Z 
3: Failed to download N11W085_15_FNF from 2015-01-01T00:00:00Z 
4: Failed to download N11W084_15_FNF from 2015-01-01T00:00:00Z 
5: /var/folders/2s/hq96vnhj3w37048l_1m9r2j00000gq/T//Rtmpc4xFEv/filede08ae0aa18.tif: No such file or directory (GDAL error 4) 
Warning messages:
1: In file.remove(unlist(download_results)) :
  cannot remove file '/var/folders/2s/hq96vnhj3w37048l_1m9r2j00000gq/T//Rtmpc4xFEv/filede08ae0aa18.tif', reason 'No such file or directory'
2: In file.remove(unlist(download_results)) :
  cannot remove file '/var/folders/2s/hq96vnhj3w37048l_1m9r2j00000gq/T//Rtmpc4xFEv/filede0823bf133c.tif', reason 'No such file or directory'
3: In file.remove(unlist(download_results)) :
  cannot remove file '/var/folders/2s/hq96vnhj3w37048l_1m9r2j00000gq/T//Rtmpc4xFEv/filede085587da79.tif', reason 'No such file or directory'
4: In file.remove(unlist(download_results)) :
  cannot remove file '/var/folders/2s/hq96vnhj3w37048l_1m9r2j00000gq/T//Rtmpc4xFEv/filede084a1869cd.tif', reason 'No such file or directory'

I know that rsi uses some rstac functionality to access the data, so I tried to download the ALOS FNF dataset using exclusively rstac to make sure there wasn't something funky happening there:

# Query for ALOS FNF data
stac_query <- rstac::stac(
  "https://planetarycomputer.microsoft.com/api/stac/v1"
)  |>
  rstac::stac_search(
    collections = "alos-fnf-mosaic",
    bbox = sf_bbox,
    datetime="2015-01-01/2021-12-31"
  ) |>
  rstac::get_request()

# Sign urls with PC authentication
signed_stac_query <- rstac::items_sign(
  stac_query,
  rstac::sign_planetary_computer()
)

# Get URLs
fnf_urls <- rstac::assets_url(signed_stac_query, asset_names = "C")
  # this returned 12 URLs. example: 
  # "https://pceo.blob.core.windows.net/palsar/v200/alos_fnf_mosaic/2020/N15W085/N11W085_20_C.tif?st=2024-07-04T17%3A33%3A46Z&se=2024-07-05T18%3A18%3A46Z&sp=rl&sv=2024-05-04&sr=c&skoid=146be268-41f7-4e92-96ad-9cf622e35904&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2024-07-05T00%3A43%3A58Z&ske=2024-07-12T00%3A43%3A58Z&sks=b&skv=2024-05-04&sig=QpISmO8vWi0lmJWVy3E5uLOmzktVHh053LArjsq3kzE%3D"

# Use `stars` package to read and plot data
library(stars)
test_fnf <- stars::read_stars(fnf_urls[1])
plot(test_fnf)

The stars package was able to read in and plot the data. I then looked into the rsi source code, found the download_function which calls to the "Failed to download" error I was seeing:

tryCatch(
  {
     sf::gdal_utils(
       "warp",
       paste0("/vsicurl/", url),
       dl_location,
       options = current_options,
       quiet = TRUE,
       config_options = gdal_config_options
      )
     },
     error = function(e) {
       rlang::warn(
         glue::glue(
           "Failed to download {items$features[[which_item]]$id %||% 'UNKNOWN'} from {items$features[[which_item]]$properties$datetime %||% 'UNKNOWN'}" # nolint
          )
         )
       download_locations[which_item, ] <- NA

I manually replicated the sf::gdal_utils function outside of my script, using the fnf_urls variable from before to see if the utility failed outside of the code. However, I was able to successfully run the gdal warps utility manually.

I'm wondering if the url variable within the source code gets impacted in some way before the gdal_utils function is called. Maybe it's something to do with the PC signing that is unique to the ALOS FNF dataset? I haven't had any other issues using get_stac_data for other PC datasets like ESA Worldcover, for example, so it's just a bit strange that it's happening with this dataset.

Let me know if there is anything you would like for me to try, and thank you so much for all the work you've put into rsi !!!

[mikemahoney218]

The issue seems to be on PC's side of things; for instance, here's a valid STAC item: https://planetarycomputer.microsoft.com/api/stac/v1/collections/alos-fnf-mosaic/items/N37W082_16_FNF

But if you scroll down to the "assets" and click the link for C, you get an error message: https://pceo.blob.core.windows.net/palsar/alos_fnf_mosaic/N37W082_16_C_F02DAR.tif

I opened a discussion on the Planetary Computer repo to see what's up, but I have no idea if/when those assets will get restored: https://github.com/microsoft/PlanetaryComputer/discussions/367

Now that said: rsi should give you the data that it could download anyway; that's why this is a warning and not an error in the code. I'll try and fix that issue.

[laurenkwick]

Ah okay, that makes sense for why I was running into some trouble. Thank you for opening that discussion!

[mikemahoney218]

I'm marking this as closed because the rsi bit of the issue -- the entire download failing when only some assets couldn't be downloaded -- is now fixed in #75. Thank you so much for opening the issue -- your examples made this one pretty easy to debug!

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.