search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Unable to Download Tile after 201904 #54

Closed prakharjhudele closed 4 years ago

prakharjhudele commented 4 years ago

Hi Chris,

I am unable to download the tiles for ctry <- "IND" after 201904

Error on console -

Please manually check for the 201907 tile at 'https://eogdata.mines.edu/download_dnb_composites_iframe.html'. If it exists please report this as a bug 2020-02-13 20:32:42: Something went wrong with the tile downloads. Aborting ...

Tile is available but package unable to download

chrisvwn commented 4 years ago

Hi,

What version of Rnightlights are you using? Please post your sessionInfo()

prakharjhudele commented 4 years ago

Please find the sessionInfo below-

R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgdal_1.4-6        sp_1.3-1           reshape2_1.4.3     lubridate_1.7.4    Rnightlights_0.2.4

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2        rstudioapi_0.10   raster_3.0-7      magrittr_1.5      hms_0.5.1         maptools_0.9-5   
 [7] maps_3.3.0        lattice_0.20-38   R6_2.4.0          rlang_0.4.1       cleangeo_0.2-2    foreach_1.4.7    
[13] stringr_1.4.0     plyr_1.8.4        fields_9.8-6      tools_3.6.1       dotCall64_1.0-0   grid_3.6.1       
[19] data.table_1.12.4 spam_2.3-0        rgeos_0.5-1       iterators_1.0.12  tibble_2.1.3      crayon_1.3.4     
[25] readr_1.3.1       vctrs_0.2.0       settings_0.2.4    codetools_0.2-16  zeallot_0.1.0     rworldmap_1.3-6  
[31] stringi_1.4.3     pillar_1.4.2      compiler_3.6.1    backports_1.1.5   foreign_0.8-71    pkgconfig_2.0.3
bedantaguru commented 4 years ago

Also looking for the solution.

chrisvwn commented 4 years ago

Everything seems to be fine with the downloads for at least 201907. Here's my output for a direct download:

Rnightlights:::downloadNlTiles("VIIRS.M", nlPeriod = "201907", tileList="00N060W") 2020-02-14 09:33:12: Downloading tile: 2019075 trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201907/vcmcfg/SVDNB_npp_20190701-20190731_00N060W_vcmcfg_v10_c201908090900.tgz' Content type 'application/x-gzip' length 522776179 bytes (498.6 MB)

Could you post the command you're running and the full output?

prakharjhudele commented 4 years ago

Hi Chris,

Thanks for your prompt responses. I have pasted below the code and output for two months. 201904 is downloaded and successful while for 201908 it fails. 

library(Rnightlights)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#> 
#>     date
library(reshape2)
library(rgdal)
#> Loading required package: sp
#> rgdal: version: 1.4-6, (SVN revision 841)
#>  Geospatial Data Abstraction Library extensions to R successfully loaded
#>  Loaded GDAL runtime: GDAL 2.2.3, released 2017/11/20
#>  Path to GDAL shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/gdal
#>  GDAL binary built with GEOS: TRUE 
#>  Loaded PROJ.4 runtime: Rel. 4.9.3, 15 August 2016, [PJ_VERSION: 493]
#>  Path to PROJ.4 shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/proj
#>  Linking to sp version: 1.3-1

#pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)
ctry <- "IND"

#ctry <- "IND" getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C://Users/Documents/Data/alan/India.zip", nlType="VIIRS.Y", nlPeriods="2015", nlStats="sum", ignoreMissing=FALSE)

#download and process monthly VIIRS stats at the highest admin level
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, 
                                      admLevel = "highest",
                                      nlType = "VIIRS.M", 
                                      nlPeriods = nlRange("201904", "201904","VIIRS.M"), 
                                      nlStats = list("sum",na.rm=TRUE),
                                      ignoreMissing=FALSE)
#> 2020-02-14 15:23:45: All stats exist
#> 2020-02-14 15:23:45: Retrieving requested data

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, 
                                      admLevel = "highest",
                                      nlType = "VIIRS.M", 
                                      nlPeriods = nlRange("201908", "201908","VIIRS.M"), 
                                      nlStats = list("sum",na.rm=TRUE),
                                      ignoreMissing=FALSE)
#> Processing missing data: IND:VIIRS.M:VCMCFG:201908:sum. This may take a while. 
#> Note: Set 'ignoreMissing=TRUE' to return only data found or 
#> 'ignoreMissing=NULL' to return NULL if not all the data is found
#> 2020-02-14 15:23:45: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> 2020-02-14 15:23:45: Downloading country polygons ...
#>   |                                                                              |                                                                      |   0%
#> 2020-02-14 15:23:45: Downloading polygon: IND
#> 2020-02-14 15:23:45: Downloading ctry poly: IND
#> 2020-02-14 15:23:45: Downloading ctry shpZip: IND
#> 2020-02-14 15:23:45: Polygon dir for IND:3.6 already exists
#>   |                                                                              |======================================================================| 100%
#> 2020-02-14 15:23:45: Downloading country polygons ... DONE
#> 2020-02-14 15:23:45: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201908****
#> 2020-02-14 15:23:45: Checking tiles required for VIIRS.M 201908
#> 2020-02-14 15:23:45: IND: Stats missing. Adding tiles
#> 2020-02-14 15:23:46: numTiles: 1, Required tiles: 75N060E
#> 2020-02-14 15:23:46: Downloading tile: 2019083
#> 2020-02-14 15:23:46: ** Tile not available on the NOAA page.
#>  Please manually check for the 201908 tile at 'https://eogdata.mines.edu/download_dnb_composites.html'. If it exists please report this as a bug **
#> 2020-02-14 15:23:46: Something went wrong with the tile downloads. Aborting ...
#> 2020-02-14 15:23:46: **** COMPLETED PROCESSING :ctryCodes=IND, admLevels=gadm36_IND_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2020-02-14 15:23:46: An error occurred

Created on 2020-02-14 by the reprex package (v0.3.0)

chrisvwn commented 4 years ago

Oh yes. There was a slight change in the iframe URL. This change has been made in the github dev version but that version is still buggy.

Try running this before re-running getCtryNlData: 

pkgOptions("ntLtsIndexUrlVIIRS.M" = "https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")
prakharjhudele commented 4 years ago

Hi ,

I tried this actually changing the iframe but still didn't work.

library(Rnightlights)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#> 
#>     date
library(reshape2)
library(rgdal)
#> Loading required package: sp
#> rgdal: version: 1.4-6, (SVN revision 841)
#>  Geospatial Data Abstraction Library extensions to R successfully loaded
#>  Loaded GDAL runtime: GDAL 2.2.3, released 2017/11/20
#>  Path to GDAL shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/gdal
#>  GDAL binary built with GEOS: TRUE 
#>  Loaded PROJ.4 runtime: Rel. 4.9.3, 15 August 2016, [PJ_VERSION: 493]
#>  Path to PROJ.4 shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/proj
#>  Linking to sp version: 1.3-1

#pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)
ctry <- "IND"

pkgOptions("ntLtsIndexUrlVIIRS.M" = "https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")

#download and process monthly VIIRS stats at the highest admin level
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, 
                                      admLevel = "highest",
                                      nlType = "VIIRS.M", 
                                      nlPeriods = nlRange("201904", "201904","VIIRS.M"), 
                                      nlStats = list("sum",na.rm=TRUE),
                                      ignoreMissing=FALSE)
#> 2020-02-14 20:58:02: All stats exist
#> 2020-02-14 20:58:02: Retrieving requested data

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, 
                                      admLevel = "highest",
                                      nlType = "VIIRS.M", 
                                      nlPeriods = nlRange("201908", "201908","VIIRS.M"), 
                                      nlStats = list("sum",na.rm=TRUE),
                                      ignoreMissing=FALSE)
#> Processing missing data: IND:VIIRS.M:VCMCFG:201908:sum. This may take a while. 
#> Note: Set 'ignoreMissing=TRUE' to return only data found or 
#> 'ignoreMissing=NULL' to return NULL if not all the data is found
#> 2020-02-14 20:58:02: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> 2020-02-14 20:58:02: Downloading country polygons ...
#>   |                                                                              |                                                                      |   0%
#> 2020-02-14 20:58:02: Downloading polygon: IND
#> 2020-02-14 20:58:02: Downloading ctry poly: IND
#> 2020-02-14 20:58:02: Downloading ctry shpZip: IND
#> 2020-02-14 20:58:02: Polygon dir for IND:3.6 already exists
#>   |                                                                              |======================================================================| 100%
#> 2020-02-14 20:58:02: Downloading country polygons ... DONE
#> 2020-02-14 20:58:02: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201908****
#> 2020-02-14 20:58:02: Checking tiles required for VIIRS.M 201908
#> 2020-02-14 20:58:02: IND: Stats missing. Adding tiles
#> 2020-02-14 20:58:03: numTiles: 1, Required tiles: 75N060E
#> 2020-02-14 20:58:03: Downloading tile: 2019083
#> 2020-02-14 20:58:04: ** Tile not available on the NOAA page.
#>  Please manually check for the 201908 tile at 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'. If it exists please report this as a bug **
#> 2020-02-14 20:58:04: Something went wrong with the tile downloads. Aborting ...
#> 2020-02-14 20:58:04: **** COMPLETED PROCESSING :ctryCodes=IND, admLevels=gadm36_IND_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2020-02-14 20:58:04: An error occurred

Created on 2020-02-14 by the reprex package (v0.3.0)

chrisvwn commented 4 years ago

Okay. Try clearing the temp cache. Run Rnightlights:::nlCleanup()

prakharjhudele commented 4 years ago

It works perfectly now!! Thanks a lot, Chris.

chrisvwn commented 4 years ago

Glad that worked. You're always welcome!