Closed prakharjhudele closed 4 years ago
Hi,
What version of Rnightlights are you using? Please post your sessionInfo()
Please find the sessionInfo below-
R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rgdal_1.4-6 sp_1.3-1 reshape2_1.4.3 lubridate_1.7.4 Rnightlights_0.2.4
loaded via a namespace (and not attached):
[1] Rcpp_1.0.2 rstudioapi_0.10 raster_3.0-7 magrittr_1.5 hms_0.5.1 maptools_0.9-5
[7] maps_3.3.0 lattice_0.20-38 R6_2.4.0 rlang_0.4.1 cleangeo_0.2-2 foreach_1.4.7
[13] stringr_1.4.0 plyr_1.8.4 fields_9.8-6 tools_3.6.1 dotCall64_1.0-0 grid_3.6.1
[19] data.table_1.12.4 spam_2.3-0 rgeos_0.5-1 iterators_1.0.12 tibble_2.1.3 crayon_1.3.4
[25] readr_1.3.1 vctrs_0.2.0 settings_0.2.4 codetools_0.2-16 zeallot_0.1.0 rworldmap_1.3-6
[31] stringi_1.4.3 pillar_1.4.2 compiler_3.6.1 backports_1.1.5 foreign_0.8-71 pkgconfig_2.0.3
Also looking for the solution.
Everything seems to be fine with the downloads for at least 201907. Here's my output for a direct download:
Rnightlights:::downloadNlTiles("VIIRS.M", nlPeriod = "201907", tileList="00N060W") 2020-02-14 09:33:12: Downloading tile: 2019075 trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201907/vcmcfg/SVDNB_npp_20190701-20190731_00N060W_vcmcfg_v10_c201908090900.tgz' Content type 'application/x-gzip' length 522776179 bytes (498.6 MB)
Could you post the command you're running and the full output?
Hi Chris,
Thanks for your prompt responses. I have pasted below the code and output for two months. 201904 is downloaded and successful while for 201908 it fails.
library(Rnightlights)
library(lubridate)
#>
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#>
#> date
library(reshape2)
library(rgdal)
#> Loading required package: sp
#> rgdal: version: 1.4-6, (SVN revision 841)
#> Geospatial Data Abstraction Library extensions to R successfully loaded
#> Loaded GDAL runtime: GDAL 2.2.3, released 2017/11/20
#> Path to GDAL shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/gdal
#> GDAL binary built with GEOS: TRUE
#> Loaded PROJ.4 runtime: Rel. 4.9.3, 15 August 2016, [PJ_VERSION: 493]
#> Path to PROJ.4 shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/proj
#> Linking to sp version: 1.3-1
#pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)
ctry <- "IND"
#ctry <- "IND" getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C://Users/Documents/Data/alan/India.zip", nlType="VIIRS.Y", nlPeriods="2015", nlStats="sum", ignoreMissing=FALSE)
#download and process monthly VIIRS stats at the highest admin level
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry,
admLevel = "highest",
nlType = "VIIRS.M",
nlPeriods = nlRange("201904", "201904","VIIRS.M"),
nlStats = list("sum",na.rm=TRUE),
ignoreMissing=FALSE)
#> 2020-02-14 15:23:45: All stats exist
#> 2020-02-14 15:23:45: Retrieving requested data
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry,
admLevel = "highest",
nlType = "VIIRS.M",
nlPeriods = nlRange("201908", "201908","VIIRS.M"),
nlStats = list("sum",na.rm=TRUE),
ignoreMissing=FALSE)
#> Processing missing data: IND:VIIRS.M:VCMCFG:201908:sum. This may take a while.
#> Note: Set 'ignoreMissing=TRUE' to return only data found or
#> 'ignoreMissing=NULL' to return NULL if not all the data is found
#> 2020-02-14 15:23:45: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> 2020-02-14 15:23:45: Downloading country polygons ...
#> | | | 0%
#> 2020-02-14 15:23:45: Downloading polygon: IND
#> 2020-02-14 15:23:45: Downloading ctry poly: IND
#> 2020-02-14 15:23:45: Downloading ctry shpZip: IND
#> 2020-02-14 15:23:45: Polygon dir for IND:3.6 already exists
#> | |======================================================================| 100%
#> 2020-02-14 15:23:45: Downloading country polygons ... DONE
#> 2020-02-14 15:23:45: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201908****
#> 2020-02-14 15:23:45: Checking tiles required for VIIRS.M 201908
#> 2020-02-14 15:23:45: IND: Stats missing. Adding tiles
#> 2020-02-14 15:23:46: numTiles: 1, Required tiles: 75N060E
#> 2020-02-14 15:23:46: Downloading tile: 2019083
#> 2020-02-14 15:23:46: ** Tile not available on the NOAA page.
#> Please manually check for the 201908 tile at 'https://eogdata.mines.edu/download_dnb_composites.html'. If it exists please report this as a bug **
#> 2020-02-14 15:23:46: Something went wrong with the tile downloads. Aborting ...
#> 2020-02-14 15:23:46: **** COMPLETED PROCESSING :ctryCodes=IND, admLevels=gadm36_IND_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2020-02-14 15:23:46: An error occurred
Created on 2020-02-14 by the reprex package (v0.3.0)
Oh yes. There was a slight change in the iframe URL. This change has been made in the github dev version but that version is still buggy.
Try running this before re-running getCtryNlData:
pkgOptions("ntLtsIndexUrlVIIRS.M" = "https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")
Hi ,
I tried this actually changing the iframe but still didn't work.
library(Rnightlights)
library(lubridate)
#>
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#>
#> date
library(reshape2)
library(rgdal)
#> Loading required package: sp
#> rgdal: version: 1.4-6, (SVN revision 841)
#> Geospatial Data Abstraction Library extensions to R successfully loaded
#> Loaded GDAL runtime: GDAL 2.2.3, released 2017/11/20
#> Path to GDAL shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/gdal
#> GDAL binary built with GEOS: TRUE
#> Loaded PROJ.4 runtime: Rel. 4.9.3, 15 August 2016, [PJ_VERSION: 493]
#> Path to PROJ.4 shared files: C:/Users/prakh/Documents/R/win-library/3.6/rgdal/proj
#> Linking to sp version: 1.3-1
#pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)
ctry <- "IND"
pkgOptions("ntLtsIndexUrlVIIRS.M" = "https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")
#download and process monthly VIIRS stats at the highest admin level
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry,
admLevel = "highest",
nlType = "VIIRS.M",
nlPeriods = nlRange("201904", "201904","VIIRS.M"),
nlStats = list("sum",na.rm=TRUE),
ignoreMissing=FALSE)
#> 2020-02-14 20:58:02: All stats exist
#> 2020-02-14 20:58:02: Retrieving requested data
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry,
admLevel = "highest",
nlType = "VIIRS.M",
nlPeriods = nlRange("201908", "201908","VIIRS.M"),
nlStats = list("sum",na.rm=TRUE),
ignoreMissing=FALSE)
#> Processing missing data: IND:VIIRS.M:VCMCFG:201908:sum. This may take a while.
#> Note: Set 'ignoreMissing=TRUE' to return only data found or
#> 'ignoreMissing=NULL' to return NULL if not all the data is found
#> 2020-02-14 20:58:02: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> 2020-02-14 20:58:02: Downloading country polygons ...
#> | | | 0%
#> 2020-02-14 20:58:02: Downloading polygon: IND
#> 2020-02-14 20:58:02: Downloading ctry poly: IND
#> 2020-02-14 20:58:02: Downloading ctry shpZip: IND
#> 2020-02-14 20:58:02: Polygon dir for IND:3.6 already exists
#> | |======================================================================| 100%
#> 2020-02-14 20:58:02: Downloading country polygons ... DONE
#> 2020-02-14 20:58:02: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201908****
#> 2020-02-14 20:58:02: Checking tiles required for VIIRS.M 201908
#> 2020-02-14 20:58:02: IND: Stats missing. Adding tiles
#> 2020-02-14 20:58:03: numTiles: 1, Required tiles: 75N060E
#> 2020-02-14 20:58:03: Downloading tile: 2019083
#> 2020-02-14 20:58:04: ** Tile not available on the NOAA page.
#> Please manually check for the 201908 tile at 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'. If it exists please report this as a bug **
#> 2020-02-14 20:58:04: Something went wrong with the tile downloads. Aborting ...
#> 2020-02-14 20:58:04: **** COMPLETED PROCESSING :ctryCodes=IND, admLevels=gadm36_IND_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
#> Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2020-02-14 20:58:04: An error occurred
Created on 2020-02-14 by the reprex package (v0.3.0)
Okay. Try clearing the temp cache. Run Rnightlights:::nlCleanup()
It works perfectly now!! Thanks a lot, Chris.
Glad that worked. You're always welcome!
Hi Chris,
I am unable to download the tiles for ctry <- "IND" after 201904
Error on console -
Please manually check for the 201907 tile at 'https://eogdata.mines.edu/download_dnb_composites_iframe.html'. If it exists please report this as a bug 2020-02-13 20:32:42: Something went wrong with the tile downloads. Aborting ...
Tile is available but package unable to download