Timeout MODIS server error when downloading

[agoodman312]

Issue description

I am experiencing a timeout error when I try to download MODIS tiles from MOD44B.006 from 2000-2020, it says Timeout is reached after 10001 milliseconds, error: http server seems to be down! try again later. Aborting! -I tried increasing the httr timeout time in the source code, following the instructions of previous posts but that did not work -I tried redoing the analysis throughout the week because I know the MODIS servers are down on wednesdays

Reproducible example

## PLEASE DELETE AND WRITE YOUR OWN
library(MODIStsp)
MODIStsp(gui = FALSE, opts_file = "/user/desktop/test.json")

test.txt

Expected and actual behavior

## PASTE HERE THE OUTPUT OF YOUR EXAMPLE CODE

GDAL version in use: 3.2.1

Listening on http://127.0.0.1:4217 old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() Reading layer OGRGeoJSON' from data source { "type": "FeatureCollection", "features": [ { "type": "Feature", "properties": { "X_leaflet_id": 1829, "layerId": "1829", "edit_id": "1829" }, "geometry": { "type": "Polygon", "coordinates": [ [ [123.948441, -37.049149], [129.925003, -35.347056], [134.671097, -38.439069], [138.362503, -41.404111], [142.405472, -44.365588], [146.448441, -45.731588], [150.667191, -44.491118], [152.249222, -40.874584], [153.479691, -37.747372], [155.764847, -33.315037], [156.643753, -29.566888], [156.467972, -25.515798], [155.589066, -21.486936], [151.897659, -18.847162], [147.678909, -13.446391], [146.800003, -12.07491], [143.811722, -9.138029], [137.659378, -8.443168], [131.155472, -8.443168], [119.378128, -12.246747], [113.753128, -17.511141], [110.061722, -21.323279], [107.952347, -27.09153], [108.655472, -33.900627], [113.577347, -38.576622], [121.663284, -39.802734], [123.948441, -37.049149] ] ] } } ] }' using driver `GeoJSON' Simple feature collection with 1 feature and 3 fields Geometry type: POLYGON Dimension: XY Bounding box: xmin: 107.9523 ymin: -45.73159 xmax: 156.6438 ymax: -8.443168 Geodetic CRS: WGS 84 old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() old-style crs object detected; please recreate object with a recent sf::st_crs() [Wed Sep 29 00:47:56 2021] MODIStsp --> Starting processing [Wed Sep 29 00:47:56 2021] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOLT/MOD44B.006/ Error in curl::curl_fetch_memory(url, handle = handle): Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 10003 milliseconds with 0 out of 0 bytes received Request failed [ERROR]. Retrying in 1 seconds... [Wed Sep 29 00:48:50 2021] Error: http server seems to be down! Please try again later. Aborting!

System information

# PASTE HERE YOUR OUTPUT

Additional context

Scientific publication

[ranghetti]

Timeout problems may depend on temporary unavailability (server side), archived products (server side) or problems with internet connection (client side).

I'm seeing products can be downloaded correctly, so I can exclude server-side problems:

GDAL version in use: 3.0.4
[Wed Sep 29 10:08:59 2021] MODIStsp --> Starting processing
[Wed Sep 29 10:08:59 2021] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD44B.006/
[Wed Sep 29 10:09:06 2021] Retrieving list of available ` Terra ` Files for Year 2000
[Wed Sep 29 10:09:17 2021] Downloading Terra Files for date: 2000_03_05 : 1 of:  32
[Wed Sep 29 10:09:18 2021] Downloading Terra Files for date: 2000_03_05 : 2 of:  32
[Wed Sep 29 10:09:21 2021] Downloading Terra Files for date: 2000_03_05 : 3 of:  32
[Wed Sep 29 10:09:47 2021] Downloading Terra Files for date: 2000_03_05 : 4 of:  32

Please try again (supposing there were temporary server-side problems, now solved) or check your internet connection / try using a different machine (supposing client-side problems).

[micha-silver]

I'm also seeing this problem. Using two different machines (with different internet connections) and thru the week. The issue was reported previously. And at that time there was a PR to increase the timeout to 20 seconds. Was this implemented? I still see a 10 sec timeout, with only 5 retries. Thanks

[ranghetti]

The PR (#220) was reverted with 06c39d33ef6d8e83a46b5e3106eab49a21c5da11 because it was causing other problems (now I did not remenber what, since I did not annontate them). If you want to change it on your side you can fork the reporisotry and change the source code similarly to 231fee340e1bbb93bca3b0c52fb7d206a1418810 . In the future I will consider to manage the timeout duration as a function argument.

[micha-silver]

body p { margin-bottom: 0cm; margin-top: 0pt; } 

Thanks very much. I will try

On 06/12/2021 11:12, Luigi Ranghetti
  wrote:

  The PR (#220) was reverted with 06c39d3 because it was
    causing other problems (now I did not remenber what, since I did
    not annontate them).
    If you want to change it on your side you can fork the
    reporisotry and change the source code similarly to 231fee3 . In the future I
    will consider to manage the timeout duration as a function
    argument.
  —
    You are receiving this because you commented.
    Reply to this email directly, view it on GitHub, or unsubscribe.
    Triage notifications on the go with GitHub Mobile for iOS or Android.

  [

{ @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/ropensci/MODIStsp/issues/243#issuecomment-986580721", "url": "https://github.com/ropensci/MODIStsp/issues/243#issuecomment-986580721", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

-- 

Micha Silver Ben Gurion Univ. Sde Boker, Remote Sensing Lab cell: +972-523-665918 https://orcid.org/0000-0002-1128-1325