search

ropensci / MODIStsp

An "R" package for automatic download and preprocessing of MODIS Land Products Time Series
https://docs.ropensci.org/MODIStsp
GNU General Public License v3.0
155 stars 51 forks source link

Cannot Download data/ downloading too slow #223

Closed ChrisMwene closed 3 years ago

ChrisMwene commented 3 years ago

Hi Lorenzo For the past 1 week now, I have been trying to download some Vegetation Indexes_16Days_250m for a 10 year period from 2010/1/1 to 2020. In some cases, downloads for a single date terra files takes more than 3 hours to complete. So far I have managed to download only the first 2 months from 2010 and restarting download from the last date is unsuccessful.

Under normal circumstances whats the average completion time when downloading terra files for a single date?

As you suggested in your comment https://github.com/ropensci/MODIStsp/issues/93#issuecomment-339695050, this is what I get when I run MODIStsp(test = 4)

MODIStsp(test = 4) MODIStsp is running in test mode. GDAL version in use: 3.0.4 [Fri Nov 27 17:13:34 2020] MODIStsp --> Starting processing [Fri Nov 27 17:13:34 2020] Accessing http server at: https://n5eil01u.ecs.nsidc.org/MOST/MOD10CM.006/ [Fri Nov 27 17:13:45 2020] Retrieving list of available Terra Files for Year 2015 [Fri Nov 27 17:13:49 2020] Downloading Terra Files for date: 2015_08_01 : 1 of: 1 [Fri Nov 27 17:14:36 2020] [Fri Nov 27 17:14:36 2020] 1 files for date: 2015.08.01 were successfully downloaded! [Fri Nov 27 17:14:37 2020] Processing and Reprojecting Terra SN_COV_MNT files for date: 2015_08_01 [Fri Nov 27 17:14:46 2020] Retrieving list of available Terra Files for Year 2016 [Fri Nov 27 17:14:48 2020] Downloading Terra Files for date: 2016_08_01 : 1 of: 1 [Fri Nov 27 17:15:39 2020] [Fri Nov 27 17:15:39 2020] 1 files for date: 2016.08.01 were successfully downloaded! [Fri Nov 27 17:15:39 2020] Processing and Reprojecting Terra SN_COV_MNT files for date: 2016_08_01 [Fri Nov 27 17:15:43 2020] Accessing http server at: https://n5eil01u.ecs.nsidc.org/MOSA/MYD10CM.006/ [Fri Nov 27 17:15:46 2020] Retrieving list of available Aqua Files for Year 2015 [Fri Nov 27 17:15:48 2020] Downloading Aqua Files for date: 2015_08_01 : 1 of: 1 [Fri Nov 27 17:16:56 2020] [Fri Nov 27 17:16:56 2020] 1 files for date: 2015.08.01 were successfully downloaded! [Fri Nov 27 17:16:56 2020] Processing and Reprojecting Aqua SN_COV_MNT files for date: 2015_08_01 [Fri Nov 27 17:17:02 2020] Retrieving list of available Aqua Files for Year 2016 [Fri Nov 27 17:17:04 2020] Downloading Aqua Files for date: 2016_08_01 : 1 of: 1 [Fri Nov 27 17:18:14 2020] [Fri Nov 27 17:18:14 2020] 1 files for date: 2016.08.01 were successfully downloaded! [Fri Nov 27 17:18:14 2020] Processing and Reprojecting Aqua SN_COV_MNT files for date: 2016_08_01 [Fri Nov 27 17:18:17 2020] Creating Virtual Files and R time series for layer SN_COV_MNT [Fri Nov 27 17:18:17 2020] Creating Virtual Files and R time series for layer SN_COV_MNT [Fri Nov 27 17:18:18 2020] Creating Virtual Files and R time series for layer SN_COV_MNT [Fri Nov 27 17:18:18 2020] Total Processing Time: 4.73123201529185 [Fri Nov 27 17:18:18 2020] MODIStsp processed files are in: C:\Users\CHRIST~1\AppData\Local\Temp\RtmpGQjZRp/MODIStsp [Fri Nov 27 17:18:18 2020] Original downloaded MODIS HDF files are in: C:\Users\CHRIST~1\AppData\Local\Temp\RtmpGQjZRp/MODIStsp/HDFs [Fri Nov 27 17:18:18 2020] Processing options saved to: C:\Users\CHRIST~1\AppData\Local\Temp\RtmpGQjZRp/MODIStsp/MODIStsp_2020-11-27.json [1] "DONE"

and MODIStsp(test = 6)

MODIStsp(test = 6) MODIStsp is running in test mode. GDAL version in use: 3.0.4 [Fri Nov 27 17:35:29 2020] MODIStsp --> Starting processing [Fri Nov 27 17:35:29 2020] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOTA/MCD15A2H.006/ [Fri Nov 27 17:35:34 2020] Retrieving list of available Combined Files for Year 2016 [Fri Nov 27 17:35:51 2020] Downloading Terra Files for date: 2016_07_19 : 1 of: 2 [Fri Nov 27 17:50:10 2020] Downloading Terra Files for date: 2016_07_19 : 2 of: 2 [Fri Nov 27 17:54:03 2020] [Fri Nov 27 17:54:03 2020] 2 files for date: 2016.07.19 were successfully downloaded! [Fri Nov 27 17:54:03 2020] Processing and Reprojecting Terra Lai files for date: 2016_07_19 [Fri Nov 27 17:54:23 2020] Downloading Terra Files for date: 2016_07_27 : 1 of: 2 [Fri Nov 27 18:07:54 2020] Downloading Terra Files for date: 2016_07_27 : 2 of: 2 [Fri Nov 27 18:10:10 2020] [Fri Nov 27 18:10:10 2020] 2 files for date: 2016.07.27 were successfully downloaded! [Fri Nov 27 18:10:10 2020] Processing and Reprojecting Terra Lai files for date: 2016_07_27 [Fri Nov 27 18:10:13 2020] Creating Virtual Files and R time series for layer Lai [Fri Nov 27 18:10:13 2020] Total Processing Time: 34.7278120001157 [Fri Nov 27 18:10:13 2020] MODIStsp processed files are in: C:\Users\CHRIST~1\AppData\Local\Temp\RtmpGQjZRp/MODIStsp [Fri Nov 27 18:10:13 2020] Original downloaded MODIS HDF files are in: C:\Users\CHRIST~1\AppData\Local\Temp\RtmpGQjZRp/MODIStsp/HDFs [Fri Nov 27 18:10:13 2020] Processing options saved to: C:\Users\CHRIST~1\AppData\Local\Temp\RtmpGQjZRp/MODIStsp/MODIStsp_2020-11-27.json [1] "DONE"

these are the options am using options.json.txt

Please help.

ranghetti commented 3 years ago

Hi, I have launched your processing: in my case, a single date takes about 3-4 minutes to be downloaded (globally, all the processing took 12 hours). Clearly this can vary depending on your internet connection speed, and servers could be sbjected to temporary drops. Have you tried to launch it from another pc / connection?

ChrisMwene commented 3 years ago

I tried using a different connection and now am able to download a single date at less than 4min average speed. Thanks for your assistance.

Chris.