Large requests cause timeout error 524 in `get_raster()`

AndreaSanchezTapia commented 11 months ago

get_raster() is failing at different combinations of spatial and temporal resolution and large areas.

tl,dr: It's a timeout problem (524) that creates an error parsing problem.

At the origin:

fishing_effort <- get_raster(spatial_resolution = 'high',
                              temporal_resolution = 'monthly',
                              group_by = 'vessel_id',
                              date_range = '2022-01-01,2022-12-31',
                              region = 8321,
                              region_source = 'eez',
                              key = gfw_auth())

Returns:

Error in `httr2::req_perform()`:
! Failed to parse error body with method defined in req_error()
Caused by error in `check_content_type()`:
! Unexpected content type 'text/html'
Expecting 'application/json'
Or suffix '+json'
ℹ Override check with `check_type = FALSE`
Run `rlang::last_trace()` to see where the error occurred.
Called from: signal_abort(cnd, .file)

When setting check_type = FALSE in gist_error_body() the error parsing changes:

Error in `httr2::req_perform()`:
! Failed to parse error body with method defined in req_error()
Caused by error:
! lexical error: invalid char in json text.
                                       <!DOCTYPE html> <!--[if lt IE 7
                     (right here) ------^
Run `rlang::last_trace()` to see where the error occurred.
Called from: signal_abort(cnd, .file)

When removing error parsing the timeout error is revealed:

response <- endpoint %>%
     httr2::req_headers(Authorization = paste("Bearer",
                                              key,
                                              sep = " "),
                        `Content-Type` = 'application/json') %>%
     httr2::req_body_raw(., body = region) %>%
     # httr2::req_error(req = ., body = gist_error_body) %>%
     httr2::req_perform(.) %>%
     httr2::resp_body_raw(.)

Error in resp_abort(resp, error_body(req, resp)) : HTTP 524 NA.

AndreaSanchezTapia commented 2 months ago

An update on Jun 26 while completing the migration to the most recent API version:

Some complex requests continue to time out and return error 524.
When this happens, the user can now access the last-report endpoint with the API token and check if any previous requests are pending.
The last-report endpoint will be implemented into gfwr soon (TBD)

AndreaSanchezTapia commented 2 months ago

Closing this issue as the endpoint and the instructions have been added to the package

GlobalFishingWatch / gfwr

Large requests cause timeout error 524 in `get_raster()` #110