tile downloading error 'A connection with the server could not be established'

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.

GNU General Public License v3.0

47 stars 14 forks source link

tile downloading error 'A connection with the server could not be established' #72

Closed sxz745 closed 3 years ago

sxz745 commented 3 years ago

Hi Chris,

I'm trying to run the example code as follows

ctry <- "KEN" #replace to run for any other country highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", nlPeriods = nlRange("201401", "201412"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

======================================= The output looks like:

2021-01-24 22:42:20: NlRange autodetected nlType: VIIRS.M Processing missing data: KEN:VIIRS.M:VCMCFG:201401:sum, VIIRS.M:VCMCFG:201402:sum, VIIRS.M:VCMCFG:201403:sum, VIIRS.M:VCMCFG:201404:sum, VIIRS.M:VCMCFG:201405:sum, VIIRS.M:VCMCFG:201406:sum, VIIRS.M:VCMCFG:201407:sum, VIIRS.M:VCMCFG:201408:sum, VIIRS.M:VCMCFG:201409:sum, VIIRS.M:VCMCFG:201410:sum, VIIRS.M:VCMCFG:201411:sum, VIIRS.M:VCMCFG:201412:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-01-24 22:42:20: START PROCESSING: ctryCodes=KEN, admLevels=list(KEN = "gadm36_KEN_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=aria, cropMaskMethod=gdal, extractMethod=rast 2021-01-24 22:42:20: Downloading country polygons ... 2021-01-24 22:42:20: Downloading polygon: KEN 2021-01-24 22:42:20: Downloading ctry poly: KEN 2021-01-24 22:42:20: Downloading ctry shpZip: KEN 2021-01-24 22:42:20: Polygon dir for KEN:3.6 already exists 2021-01-24 22:42:20: Downloading country polygons ... DONE 2021-01-24 22:42:20: PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201401 2021-01-24 22:42:20: Checking tiles required for VIIRS.M 201401 2021-01-24 22:42:20: KEN: Stats missing. Adding tiles 2021-01-24 22:42:20: numTiles: 2, Required tiles: 75N060W,00N060W 2021-01-24 22:42:20: Downloading tile: 2014012

then comes the error:

trying URL 'https://eogdata.mines.edu/download_dnb_composites_iframe.html' Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : cannot open URL 'https://eogdata.mines.edu/download_dnb_composites_iframe.html' In addition: Warning message: In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : InternetOpenUrl failed: 'A connection with the server could not be established'

Issue #63 looks similar to this Error in some sense but I did not manage to solve this. Have you encountered such problems before? I have RStudio with R 3.6.3, curl package of version 4.3, Rnightlight package of version 0.2.4 (also tried 0.2.0 but has the same issue).

Thank you in advance.

chrisvwn commented 3 years ago

Hi @sxz745 . This looks like a temporary problem. For some reason the webpage could not be loaded. Most likely trying again will solve this However, there have been changes to the download site e.g. it requires authentication. I suggest installing the github version via: devtools::install_github("chrisvwn/Rnightlights")

sxz745 commented 3 years ago

Hi Chris, thanks for the reply. I've tried several times but still keep getting this error. Then I tried remove the existing package and reinstalled the GitHub version as you recommended. Now I get the following error message when I run exactly the same code:

Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : Invalid extension(s) supplied

I've searched but found little information regarding this error message - Invalid extension(s) supplied. Do you have any ideas what the problem might be?

chrisvwn commented 3 years ago

Hi,

There are some changes to getCtryNlData. In this case you need to choose the configName and extension to work on which determines which image is downloaded and extracted.

Try running this:

ctry <- getCtryNlData(ctryCode = "KEN", admLevel = "highest", nlTypes = "VIIRS.M", nlPeriods = nlRange("201401", "201412"), nlStats = list("sum", na.rm=TRUE), configNames = "vcmcfg", extensions = "avg_rade9", removeGasFlaresMethod = "VTM")

sxz745 commented 3 years ago

This is helpful. Thanks. Now I'm able to download the tiles. However, a new error message appeared:

Error in availUrls[[nlType]][[configName]] : subscript out of bounds

So the process stopped here and no nightlight data is printed in the data csv file. Part of the output is presented below.

================================================ 2021-02-19 10:51:33: Successfully retrieved access token trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201401/vcmcfg/SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.tgz' Content type 'application/x-gzip' length 565915974 bytes (539.7 MB) downloaded 539.7 MB

2021-02-19 10:52:12: Extracting C:\Users\dell\AppData\Local\Temp\RtmpyCrDrp/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.tgz 2021-02-19 10:52:12: Getting list of files in C:\Users\dell\AppData\Local\Temp\RtmpyCrDrp/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.tgz 2021-02-19 10:52:34: Extracting SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.avg_rade9h.tif 2021-02-19 10:53:20: Processing ctryCode: AFG 2021-02-19 10:53:20: Processing ctryCode: AFG AdmLevel: gadm36_AFG_1 2021-02-19 10:53:20: ProcessNlCountry: ctryCode=AFG, admLevel=gadm36_AFG_1, nlType=VIIRS.M, configName=VCMCFG, extension=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriod=201401, nlStats=list(list("sum", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL**** 2021-02-19 10:53:20: Check for existing data file 2021-02-19 10:53:20: Data file not found. Creating ... 2021-02-19 10:53:20: Data file not found. Creating ... DONE 2021-02-19 10:53:20: Begin processing 201401 2021-02-19 10:53:20: Country output raster not found. Creating 2021-02-19 10:53:20: Load polygon layer for crop 2021-02-19 10:53:20: Reading in the raster tiles 2021-02-19 10:53:20: Cropping the raster tiles 2021-02-19 10:53:20: Cropping tile = 75N060E 2021-02-19 10:53:21: Masking the raster 2021-02-19 10:53:21: Convert to raster object using fasterize 2021-02-19 10:53:22: Masking 2021-02-19 10:53:22: Writing the raster to disk 2021-02-19 10:53:23: Crop and mask using rasterize ... Done 2021-02-19 10:53:23: Processing gas-flare/background removal trying URL 'https://eogdata.mines.edu/dmsp/downloadV4composites.html' Content type 'text/html; charset=UTF-8' length 421 bytes downloaded 421 bytes

trying URL 'https://ngdc.noaa.gov/eog/viirs/download_ut_mos_tile_iframe.html' Content type 'text/html; charset=UTF-8' length unknown downloaded 224 KB

trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' Content type 'text/html; charset=UTF-8' length 270931 bytes (264 KB) downloaded 264 KB

================================================================ Then comes the error message Error in availUrls[[nlType]][[configName]] : subscript out of bounds

chrisvwn commented 3 years ago

Please post the command you're running and the whole output.

sxz745 commented 3 years ago

Please see below.

ctry <- getCtryNlData(ctryCode = "AFG",

admLevel = "highest",

nlTypes = "VIIRS.M",

nlPeriods = nlRange("201401", "201412"),

nlStats = list("sum", na.rm=TRUE),

configNames = "vcmcfg",

extensions = "avg_rade9",

removeGasFlaresMethod = "VTM") 2021-02-22 10:41:55: NlRange autodetected nlType: VIIRS.M 2021-02-22 10:41:55: Downloading ctry poly: AFG 2021-02-22 10:41:55: Downloading ctry shpZip: AFG 2021-02-22 10:41:55: Downloading https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_AFG_shp.zip trying URL 'https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_AFG_shp.zip' Content type 'application/zip' length 509958 bytes (498 KB) downloaded 498 KB

OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA.Rnightlights\polygons\POLY_AFG_GADM-3.6-SHPZIP", layer: "gadm36_AFG_0" with 1 features It has 2 fields 2021-02-22 10:42:01: Creating integer zone attribute col for polygon 2021-02-22 10:42:01: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA.Rnightlights\polygons\POLY_AFG_GADM-3.6-SHPZIP", layer: "gadm36_AFG_1" with 34 features It has 10 fields 2021-02-22 10:42:01: Creating integer zone attribute col for polygon 2021-02-22 10:42:01: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA.Rnightlights\polygons\POLY_AFG_GADM-3.6-SHPZIP", layer: "gadm36_AFG_2" with 328 features It has 13 fields 2021-02-22 10:42:02: Creating integer zone attribute col for polygon 2021-02-22 10:42:02: Writing layer with new idx col 2021-02-22 10:42:02: Saving shapefile as RDS for faster access 2021-02-22 10:42:02: Getting admLevels in AFG 2021-02-22 10:42:02: Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA.Rnightlights\polygons\POLY_AFG_GADM-3.6-SHPZIP", layer: "gadm36_AFG_0" with 1 features It has 3 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA.Rnightlights\polygons\POLY_AFG_GADM-3.6-SHPZIP", layer: "gadm36_AFG_1" with 34 features It has 11 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA.Rnightlights\polygons\POLY_AFG_GADM-3.6-SHPZIP", layer: "gadm36_AFG_2" with 328 features It has 14 fields 2021-02-22 10:42:03: Saving admLevel polygons as RDS 2021-02-22 10:42:03: Saving country admLevel structure to CSV 2021-02-22 10:42:03: ctryStruct not found. Creating Processing missing data: AFG:VIIRS.M:VCMCFG:avg_rade9:201401:sum, VIIRS.M:VCMCFG:avg_rade9:201402:sum, VIIRS.M:VCMCFG:avg_rade9:201403:sum, VIIRS.M:VCMCFG:avg_rade9:201404:sum, VIIRS.M:VCMCFG:avg_rade9:201405:sum, VIIRS.M:VCMCFG:avg_rade9:201406:sum, VIIRS.M:VCMCFG:avg_rade9:201407:sum, VIIRS.M:VCMCFG:avg_rade9:201408:sum, VIIRS.M:VCMCFG:avg_rade9:201409:sum, VIIRS.M:VCMCFG:avg_rade9:201410:sum, VIIRS.M:VCMCFG:avg_rade9:201411:sum, VIIRS.M:VCMCFG:avg_rade9:201412:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-02-22 10:42:04: START PROCESSING: ctryCodes=AFG, admLevels=list(AFG = "gadm36_AFG_1"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2021-02-22 10:42:04: Downloading country polygons ... | | 0%2021-02-22 10:42:04: Downloading polygon: AFG 2021-02-22 10:42:04: Downloading ctry poly: AFG 2021-02-22 10:42:04: Downloading ctry shpZip: AFG 2021-02-22 10:42:04: Polygon dir for AFG:3.6 already exists |===================================================================================================================================| 100% 2021-02-22 10:42:04: Downloading country polygons ... DONE 2021-02-22 10:42:04: PROCESSING nlType:VIIRS.M | configName: VCMCFG | extension: avg_rade9 | nlPeriod:201401 2021-02-22 10:42:04: Checking tiles required for VIIRS.M 201401 2021-02-22 10:42:04: AFG: Stats missing. Adding tiles 2021-02-22 10:42:04: numTiles: 1, Required tiles: 75N060E 2021-02-22 10:42:04: Downloading tile (1/1): 2014013 trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' Content type 'text/html; charset=UTF-8' length 270931 bytes (264 KB) downloaded 264 KB

2021-02-22 10:42:09: Download token expired. Refresh token expired. Requesting new 2021-02-22 10:42:09: EOG credential file not found Invalid EOG credentials Registration required for VIIRS satellite imagery access

Please browse to https://eogauth.mines.edu/auth/realms/master/account/ and register an account.

If you already have the credentials use the menu below to save the username and password.

Enter 0 to Exit.

1: Save credentials 2: View saved credentials 3: Open EOG registration page in default browser

Selection: 0 2021-02-22 10:42:49: Successfully retrieved access token trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201401/vcmcfg/SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.tgz' Content type 'application/x-gzip' length 565915974 bytes (539.7 MB) downloaded 539.7 MB

2021-02-22 10:43:27: Extracting C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.tgz 2021-02-22 10:43:27: Getting list of files in C:\Users\dell\AppData\Local\Temp\Rtmpa4DhcA/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.tgz 2021-02-22 10:43:49: Extracting SVDNB_npp_20140101-20140131_75N060E_vcmcfg_v10_c201506171538.avg_rade9h.tif 2021-02-22 10:44:33: Processing ctryCode: AFG 2021-02-22 10:44:33: Processing ctryCode: AFG AdmLevel: gadm36_AFG_1 2021-02-22 10:44:33: ProcessNlCountry: ctryCode=AFG, admLevel=gadm36_AFG_1, nlType=VIIRS.M, configName=VCMCFG, extension=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriod=201401, nlStats=list(list("sum", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL**** 2021-02-22 10:44:33: Check for existing data file 2021-02-22 10:44:33: Data file not found. Creating ... 2021-02-22 10:44:34: Data file not found. Creating ... DONE 2021-02-22 10:44:34: Begin processing 201401 2021-02-22 10:44:34: Country output raster not found. Creating 2021-02-22 10:44:34: Load polygon layer for crop 2021-02-22 10:44:34: Reading in the raster tiles 2021-02-22 10:44:34: Cropping the raster tiles 2021-02-22 10:44:34: Cropping tile = 75N060E 2021-02-22 10:44:35: Masking the raster 2021-02-22 10:44:35: Convert to raster object using fasterize 2021-02-22 10:44:36: Masking 2021-02-22 10:44:36: Writing the raster to disk 2021-02-22 10:44:37: Crop and mask using rasterize ... Done 2021-02-22 10:44:37: Processing gas-flare/background removal trying URL 'https://eogdata.mines.edu/dmsp/downloadV4composites.html' Content type 'text/html; charset=UTF-8' length 421 bytes downloaded 421 bytes

trying URL 'https://ngdc.noaa.gov/eog/viirs/download_ut_mos_tile_iframe.html' Content type 'text/html; charset=UTF-8' length unknown downloaded 228 KB

trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' Content type 'text/html; charset=UTF-8' length 270931 bytes (264 KB) downloaded 264 KB

================================================================ Then comes the error message Error in availUrls[[nlType]][[configName]] : subscript out of bounds

chrisvwn commented 3 years ago

I still cannot figure out what is going wrong. I have tested it here and it seems fine. The output from my run is below:

> ctry <- getCtryNlData(ctryCode = "KEN", admLevel = "highest", nlTypes = "VIIRS.M", nlPeriods = nlRange("201401", "201412"), nlStats = list("sum", na.rm=TRUE), configNames = "vcmcfg", extensions = "avg_rade9", removeGasFlaresMethod = "VTM")
2021-02-24 09:18:26: NlRange autodetected nlType: VIIRS.M
Processing missing data: KEN:VIIRS.M:VCMCFG:avg_rade9:201401:sum, VIIRS.M:VCMCFG:avg_rade9:201402:sum, VIIRS.M:VCMCFG:avg_rade9:201403:sum, VIIRS.M:VCMCFG:avg_rade9:201404:sum, VIIRS.M:VCMCFG:avg_rade9:201405:sum, VIIRS.M:VCMCFG:avg_rade9:201406:sum, VIIRS.M:VCMCFG:avg_rade9:201407:sum, VIIRS.M:VCMCFG:avg_rade9:201408:sum, VIIRS.M:VCMCFG:avg_rade9:201409:sum, VIIRS.M:VCMCFG:avg_rade9:201410:sum, VIIRS.M:VCMCFG:avg_rade9:201411:sum, VIIRS.M:VCMCFG:avg_rade9:201412:sum. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2021-02-24 09:18:28: **** START PROCESSING: ctryCodes=KEN, admLevels=list(KEN = "gadm36_KEN_1"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2021-02-24 09:18:28: Downloading country polygons ...
  |                                                                                                                     |   0%2021-02-24 09:18:28: Downloading polygon: KEN
2021-02-24 09:18:28: Downloading ctry poly: KEN
2021-02-24 09:18:28: Downloading ctry shpZip: KEN
2021-02-24 09:18:28: Polygon dir for KEN:3.6 already exists
  |=====================================================================================================================| 100%
2021-02-24 09:18:28: Downloading country polygons ... DONE
2021-02-24 09:18:29: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | extension: avg_rade9 | nlPeriod:201401****
2021-02-24 09:18:29: Checking tiles required for VIIRS.M 201401
2021-02-24 09:18:29: KEN: Stats missing. Adding tiles
2021-02-24 09:18:31: numTiles: 2, Required tiles: 75N060W,00N060W
2021-02-24 09:18:31: Downloading tile (1/2): 2014012
2021-02-24 09:18:35: Download token expired. Refresh token expired. Requesting new
*   Trying 138.67.208.190:443...
*   Trying 64:ff9b::8a43:d0be:443...
* Immediate connect fail for 64:ff9b::8a43:d0be: Network is unreachable
* Connected to eogauth.mines.edu (138.67.208.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; postalCode=80401; ST=Colorado; L=Golden; street=1500 Illinois Street; O=Colorado School of Mines; OU=Earth Observation Group; CN=eogauth.mines.edu
*  start date: Sep 22 00:00:00 2020 GMT
*  expire date: Oct 23 23:59:59 2021 GMT
*  subjectAltName: host "eogauth.mines.edu" matched cert's "eogauth.mines.edu"
*  issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55894aafc0e0)
> POST /auth/realms/master/protocol/openid-connect/token HTTP/2
Host: eogauth.mines.edu
accept: */*
content-type: application/x-www-form-urlencoded
content-length: 146

* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200 
< cache-control: no-store
< set-cookie: KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; Secure; HttpOnly
< set-cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; Secure; HttpOnly
< x-xss-protection: 1; mode=block
< pragma: no-cache
< x-frame-options: SAMEORIGIN
< referrer-policy: no-referrer
< date: Wed, 24 Feb 2021 06:18:37 GMT
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< content-type: application/json
< content-length: 2234
< 
* Connection #0 to host eogauth.mines.edu left intact
2021-02-24 09:18:37: Successfully retrieved access token
trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201401/vcmcfg/SVDNB_npp_20140101-20140131_75N060W_vcmcfg_v10_c201506171538.tgz'
Content type 'application/x-gzip' length 611262356 bytes (582.9 MB)
==================================================
downloaded 582.9 MB

2021-02-24 09:26:28: Extracting /media/NewVolume/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_75N060W_vcmcfg_v10_c201506171538.tgz
2021-02-24 09:26:29: Getting list of files in /media/NewVolume/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_75N060W_vcmcfg_v10_c201506171538.tgz
2021-02-24 09:27:25: Extracting SVDNB_npp_20140101-20140131_75N060W_vcmcfg_v10_c201506171538.avg_rade9h.tif
2021-02-24 09:32:28: Downloading tile (2/2): 2014015
2021-02-24 09:32:30: Download token available and not expired
trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201401/vcmcfg/SVDNB_npp_20140101-20140131_00N060W_vcmcfg_v10_c201506171538.tgz'
Content type 'application/x-gzip' length 324844599 bytes (309.8 MB)
==================================================
downloaded 309.8 MB

2021-02-24 09:36:39: Extracting /media/NewVolume/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_00N060W_vcmcfg_v10_c201506171538.tgz
2021-02-24 09:36:40: Getting list of files in /media/NewVolume/.Rnightlights/tiles/SVDNB_npp_20140101-20140131_00N060W_vcmcfg_v10_c201506171538.tgz
2021-02-24 09:37:27: Extracting SVDNB_npp_20140101-20140131_00N060W_vcmcfg_v10_c201506171538.avg_rade9h.tif
2021-02-24 09:41:46: Processing ctryCode: KEN
2021-02-24 09:41:46: Processing ctryCode: KEN AdmLevel: gadm36_KEN_1
2021-02-24 09:41:46: ProcessNlCountry: ctryCode=KEN, admLevel=gadm36_KEN_1, nlType=VIIRS.M, configName=VCMCFG, extension=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriod=201401, nlStats=list(list("sum", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2021-02-24 09:41:46: Check for existing data file
2021-02-24 09:41:46: Data file found: /media/NewVolume/.Rnightlights/data/NL_DATA_KEN_ADM1_GADM-3.6-SHPZIP.csv
2021-02-24 09:41:46: Processing stats: sum(na.rm=TRUE)
2021-02-24 09:41:46: Load country data file
2021-02-24 09:41:47: Load country polygon admin level
2021-02-24 09:41:47: Begin processing 201401
2021-02-24 09:41:47: Country output raster not found. Creating
2021-02-24 09:41:47: Load polygon layer for crop
2021-02-24 09:41:48: Reading in the raster tiles
2021-02-24 09:41:49: Cropping the raster tiles 
2021-02-24 09:41:49: Cropping tile = 75N060W
  |=====================================================================================================================| 100%

2021-02-24 09:41:52: Cropping the raster tiles 
2021-02-24 09:41:52: Cropping tile = 00N060W
  |=====================================================================================================================| 100%

2021-02-24 09:41:55: Masking the raster 
2021-02-24 09:41:55: Convert to raster object using fasterize 
2021-02-24 09:41:57: Masking 
2021-02-24 09:41:58: Writing the raster to disk 
  |=====================================================================================================================| 100%

2021-02-24 09:41:59: Crop and mask using rasterize ... Done
2021-02-24 09:41:59: Processing gas-flare/background removal
trying URL 'https://eogdata.mines.edu/dmsp/downloadV4composites.html'
Content type 'text/html; charset=UTF-8' length 421 bytes
==================================================
downloaded 421 bytes

trying URL 'https://ngdc.noaa.gov/eog/viirs/download_ut_mos_tile_iframe.html'
downloaded 228 KB

2021-02-24 09:42:08: RGF: /media/NewVolume/.Rnightlights/outputrasters/NL_GFMask_KEN_VIIRS.Y_2016_3.6-SHPZIP.tif already exists
2021-02-24 09:42:09: Begin extracting data from the raster 
  |=====================================================================================================================| 100%
2021-02-24 09:42:18: Saving nlStat 'sum(na.rm=TRUE)' ...
2021-02-24 09:42:18: sum(na.rm=TRUE) already saved
2021-02-24 09:42:18: Saving nlStat 'sum' ... DONE
2021-02-24 09:42:18: DONE processing KEN 201401
2021-02-24 09:42:18: **COMPLETE. Writing data to disk

Maybe try clearing the temp cache. Run Rnightlights:::nlCleanup() then try again.

sxz745 commented 3 years ago

Hi Chris, I've tried the code on three machines and I'm always getting different errors. May I have a look at your sessionInfo() please? Thanks.

chrisvwn commented 3 years ago

Hi. My sessionInfo:

R version 4.0.3 (2020-10-10)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 33 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libflexiblas.so.3.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Rnightlights_0.3.0.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5        ff_4.0.4          lattice_0.20-41   codetools_0.2-16  snow_0.4-3        digest_0.6.27    
 [7] foreach_1.5.0     grid_4.0.3        magrittr_2.0.1    settings_0.2.4    stringi_1.5.3     data.table_1.13.2
[13] raster_3.3-13     sp_1.4-4          rgdal_1.5-18      iterators_1.0.12  tools_4.0.3       stringr_1.4.0    
[19] bit_4.0.4         parallel_4.0.3    yaml_2.2.1        compiler_4.0.3    doSNOW_1.0.18

chrisvwn commented 3 years ago

Could you post your sessionInfo? In the meantime, I will install the package in a fresh environment and test it.

sxz745 commented 3 years ago

Hi Chris, I'm trying to replicate your setting on another desktop and get the following error message.

ctry <- getCtryNlData(ctryCode = "KEN", admLevel = "highest", nlTypes = "VIIRS.M", nlPeriods = nlRange("201401", "201412"), nlStats = list("sum", na.rm=TRUE), configNames = "vcmcfg", extensions = "avg_rade9", removeGasFlaresMethod = "VTM") 2021-02-27 19:56:51: NlRange autodetected nlType: VIIRS.M Processing missing data: KEN:VIIRS.M:VCMCFG:avg_rade9:201401:sum, VIIRS.M:VCMCFG:avg_rade9:201402:sum, VIIRS.M:VCMCFG:avg_rade9:201403:sum, VIIRS.M:VCMCFG:avg_rade9:201404:sum, VIIRS.M:VCMCFG:avg_rade9:201405:sum, VIIRS.M:VCMCFG:avg_rade9:201406:sum, VIIRS.M:VCMCFG:avg_rade9:201407:sum, VIIRS.M:VCMCFG:avg_rade9:201408:sum, VIIRS.M:VCMCFG:avg_rade9:201409:sum, VIIRS.M:VCMCFG:avg_rade9:201410:sum, VIIRS.M:VCMCFG:avg_rade9:201411:sum, VIIRS.M:VCMCFG:avg_rade9:201412:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-02-27 19:56:51: START PROCESSING: ctryCodes=KEN, admLevels=list(KEN = "gadm36_KEN_1"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2021-02-27 19:56:51: Downloading country polygons ... | | 0%2021-02-27 19:56:51: Downloading polygon: KEN 2021-02-27 19:56:51: Downloading ctry poly: KEN 2021-02-27 19:56:51: Downloading ctry shpZip: KEN 2021-02-27 19:56:51: Polygon dir for KEN:3.6 already exists |========================================================================| 100% 2021-02-27 19:56:51: Downloading country polygons ... DONE 2021-02-27 19:56:51: PROCESSING nlType:VIIRS.M | configName: VCMCFG | extension: avg_rade9 | nlPeriod:201401 2021-02-27 19:56:51: Checking tiles required for VIIRS.M 201401 2021-02-27 19:56:51: KEN: Stats missing. Adding tiles 2021-02-27 19:56:52: numTiles: 2, Required tiles: 75N060W,00N060W 2021-02-27 19:56:52: Downloading tile (1/2): 2014012 Error in if (!file.exists(ntLtsPageLocalName) || file.size(ntLtsPageLocalName) == : missing value where TRUE/FALSE needed

The sessionInfo() prints

sessionInfo()

R version 4.0.3 (2020-10-10) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18363) Matrix products: default locale: [1] LC_COLLATE=Chinese (Simplified)_China.936 [2] LC_CTYPE=Chinese (Simplified)_China.936
[3] LC_MONETARY=Chinese (Simplified)_China.936 [4] LC_NUMERIC=C
[5] LC_TIME=Chinese (Simplified)_China.936

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] Rnightlights_0.3.0.0 reshape2_1.4.4 lubridate_1.7.10

loaded via a namespace (and not attached): [1] ff_4.0.4 Rcpp_1.0.6 lattice_0.20-41 codetools_0.2-16 [5] foreach_1.5.1 R6_2.5.0 grid_4.0.3 plyr_1.8.6
[9] magrittr_2.0.1 httr_1.4.2 settings_0.2.4 stringi_1.5.3
[13] curl_4.3 data.table_1.14.0 raster_3.4-5 sp_1.4-5
[17] generics_0.1.0 rgdal_1.5-23 iterators_1.0.13 tools_4.0.3
[21] stringr_1.4.0 bit_4.0.4 compiler_4.0.3 rgeos_0.5-5

chrisvwn commented 3 years ago

This is a different error. Very strange. Is it always stopping with the same error now?

sxz745 commented 3 years ago

Yes. The Session which produces persistent Error in availUrls[[nlType]][[configName]] : subscript out of bounds has the following session info.

sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale: [1] LC_COLLATE=English_United Kingdom.1252 [2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] reshape2_1.4.4 lubridate_1.7.10 Rnightlights_0.3.0.0

loaded via a namespace (and not attached): [1] httr_1.4.2 pkgload_1.2.0 maps_3.3.0
[4] jsonlite_1.7.2 foreach_1.5.1 dotCall64_1.0-1
[7] assertthat_0.2.1 sp_1.4-5 rworldmap_1.3-6
[10] selectr_0.4-2 remotes_2.2.0 sessioninfo_1.1.1 [13] pillar_1.5.0 lattice_0.20-38 glue_1.4.2
[16] rvest_0.3.6 plyr_1.8.6 pkgconfig_2.0.3
[19] devtools_2.3.2 raster_3.4-5 purrr_0.3.4
[22] fasterize_1.0.3 processx_3.4.5 ff_4.0.4
[25] tibble_3.1.0 generics_0.1.0 usethis_2.0.1
[28] ellipsis_0.3.1 cachem_1.0.4 withr_2.4.1
[31] cli_2.3.1 magrittr_2.0.1 crayon_1.4.1
[34] memoise_2.0.0 maptools_1.0-2 settings_0.2.4
[37] ps_1.5.0 fs_1.5.0 fansi_0.4.2
[40] cleangeo_0.2-3 xml2_1.3.2 foreign_0.8-75
[43] class_7.3-15 pkgbuild_1.2.0 tools_3.6.3
[46] data.table_1.14.0 prettyunits_1.1.1 hms_1.0.0
[49] lifecycle_1.0.0 stringr_1.4.0 callr_3.5.1
[52] compiler_3.6.3 e1071_1.7-4 rlang_0.4.10
[55] classInt_0.4-3 units_0.7-0 grid_3.6.3
[58] RCurl_1.98-1.2 iterators_1.0.13 spam_2.6-0
[61] bitops_1.0-6 tcltk_3.6.3 testthat_3.0.2
[64] codetools_0.2-16 DBI_1.1.1 curl_4.3
[67] R6_2.5.0 dplyr_1.0.4 rgdal_1.5-23
[70] fastmap_1.1.0 rgeos_0.5-5 bit_4.0.4
[73] utf8_1.1.4 rprojroot_2.0.2 KernSmooth_2.23-16 [76] readr_1.4.0 desc_1.2.0 stringi_1.5.3
[79] Rcpp_1.0.6 fields_11.6 vctrs_0.3.6
[82] sf_0.9-7 tidyselect_1.1.0

I can't really tell what went wrong. I tried the code for another country, e.g., YEM, but it still did't work and stopped with a different error message.

ctry <- getCtryNlData(ctryCode = "YEM", admLevel = "highest", nlTypes = "VIIRS.M", nlPeriods = nlRange("201401", "201412"), nlStats = list("sum", na.rm=TRUE), configNames = "vcmcfg", extensions = "avg_rade9", removeGasFlaresMethod = "VTM")

2021-03-03 11:39:04: NlRange autodetected nlType: VIIRS.M 2021-03-03 11:39:04: Downloading ctry poly: YEM 2021-03-03 11:39:04: Downloading ctry shpZip: YEM 2021-03-03 11:39:04: Downloading https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_YEM_shp.zip trying URL 'https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_YEM_shp.zip' Content type 'application/zip' length 1332199 bytes (1.3 MB) downloaded 1.3 MB

OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_0" with 1 features It has 2 fields 2021-03-03 11:39:09: Creating integer zone attribute col for polygon 2021-03-03 11:39:09: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_1" with 21 features It has 10 fields 2021-03-03 11:39:10: Creating integer zone attribute col for polygon 2021-03-03 11:39:10: Writing layer with new idx col Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_2" with 333 features It has 13 fields 2021-03-03 11:39:11: Creating integer zone attribute col for polygon 2021-03-03 11:39:11: Writing layer with new idx col Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. 2021-03-03 11:39:11: Saving shapefile as RDS for faster access 2021-03-03 11:39:11: Getting admLevels in YEM 2021-03-03 11:39:11: Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_0" with 1 features It has 3 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_1" with 21 features It has 11 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_2" with 333 features It has 14 fields 2021-03-03 11:39:13: Saving admLevel polygons as RDS 2021-03-03 11:39:13: Saving country admLevel structure to CSV 2021-03-03 11:39:13: ctryStruct not found. Creating Processing missing data: YEM:VIIRS.M:VCMCFG:avg_rade9:201401:sum, VIIRS.M:VCMCFG:avg_rade9:201402:sum, VIIRS.M:VCMCFG:avg_rade9:201403:sum, VIIRS.M:VCMCFG:avg_rade9:201404:sum, VIIRS.M:VCMCFG:avg_rade9:201405:sum, VIIRS.M:VCMCFG:avg_rade9:201406:sum, VIIRS.M:VCMCFG:avg_rade9:201407:sum, VIIRS.M:VCMCFG:avg_rade9:201408:sum, VIIRS.M:VCMCFG:avg_rade9:201409:sum, VIIRS.M:VCMCFG:avg_rade9:201410:sum, VIIRS.M:VCMCFG:avg_rade9:201411:sum, VIIRS.M:VCMCFG:avg_rade9:201412:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-03-03 11:39:14: START PROCESSING: ctryCodes=YEM, admLevels=list(YEM = "gadm36_YEM_1"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2021-03-03 11:39:14: Downloading country polygons ... | | 0%2021-03-03 11:39:14: Downloading polygon: YEM 2021-03-03 11:39:14: Downloading ctry poly: YEM 2021-03-03 11:39:14: Downloading ctry shpZip: YEM 2021-03-03 11:39:14: Polygon dir for YEM:3.6 already exists |=================================================================| 100% 2021-03-03 11:39:14: Downloading country polygons ... DONE 2021-03-03 11:39:14: PROCESSING nlType:VIIRS.M | configName: VCMCFG | extension: avg_rade9 | nlPeriod:201401 2021-03-03 11:39:14: Checking tiles required for VIIRS.M 201401 2021-03-03 11:39:14: YEM: Stats missing. Adding tiles 2021-03-03 11:39:14: numTiles: 1, Required tiles: 75N060W 2021-03-03 11:39:14: Downloading tile (1/1): 2014012 2021-03-03 11:39:14: Tif file exists, set Overwrite = TRUE to overwrite Error in message(Sys.time(), ": Extracting ", ntLtsZipLocalNamePathVIIRS) : object 'ntLtsZipLocalNamePathVIIRS' not found

chrisvwn commented 3 years ago

Hi,

I am not sure what the problem is. It could be that I am using a later version of the package as I have made some changes locally. Unfortunately, I am having trouble logging into my github account so I cannot update the code right away.

I will try to use the package as it is on github and see if I get the same errors.

On Wed, Mar 3, 2021 at 10:01 AM sxz745 notifications@github.com wrote:

Yes. The Session which produces persistent Error in availUrls[[nlType]][[configName]] : subscript out of bounds has the following session info.

sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale: [1] LC_COLLATE=English_United Kingdom.1252 [2] LC_CTYPE=English_United Kingdom.1252 [3] LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C [5] LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] reshape2_1.4.4 lubridate_1.7.10 Rnightlights_0.3.0.0

loaded via a namespace (and not attached): [1] httr_1.4.2 pkgload_1.2.0 maps_3.3.0 [4] jsonlite_1.7.2 foreach_1.5.1 dotCall64_1.0-1 [7] assertthat_0.2.1 sp_1.4-5 rworldmap_1.3-6 [10] selectr_0.4-2 remotes_2.2.0 sessioninfo_1.1.1 [13] pillar_1.5.0 lattice_0.20-38 glue_1.4.2 [16] rvest_0.3.6 plyr_1.8.6 pkgconfig_2.0.3 [19] devtools_2.3.2 raster_3.4-5 purrr_0.3.4 [22] fasterize_1.0.3 processx_3.4.5 ff_4.0.4 [25] tibble_3.1.0 generics_0.1.0 usethis_2.0.1 [28] ellipsis_0.3.1 cachem_1.0.4 withr_2.4.1 [31] cli_2.3.1 magrittr_2.0.1 crayon_1.4.1 [34] memoise_2.0.0 maptools_1.0-2 settings_0.2.4 [37] ps_1.5.0 fs_1.5.0 fansi_0.4.2 [40] cleangeo_0.2-3 xml2_1.3.2 foreign_0.8-75 [43] class_7.3-15 pkgbuild_1.2.0 tools_3.6.3 [46] data.table_1.14.0 prettyunits_1.1.1 hms_1.0.0 [49] lifecycle_1.0.0 stringr_1.4.0 callr_3.5.1 [52] compiler_3.6.3 e1071_1.7-4 rlang_0.4.10 [55] classInt_0.4-3 units_0.7-0 grid_3.6.3 [58] RCurl_1.98-1.2 iterators_1.0.13 spam_2.6-0 [61] bitops_1.0-6 tcltk_3.6.3 testthat_3.0.2 [64] codetools_0.2-16 DBI_1.1.1 curl_4.3 [67] R6_2.5.0 dplyr_1.0.4 rgdal_1.5-23 [70] fastmap_1.1.0 rgeos_0.5-5 bit_4.0.4 [73] utf8_1.1.4 rprojroot_2.0.2 KernSmooth_2.23-16 [76] readr_1.4.0 desc_1.2.0 stringi_1.5.3 [79] Rcpp_1.0.6 fields_11.6 vctrs_0.3.6 [82] sf_0.9-7 tidyselect_1.1.0

I can't really tell what went wrong. I tried the code for another country, e.g., YEM, but it still did't work and stopped with a different error message.

ctry <- getCtryNlData(ctryCode = "YEM",

-
                 admLevel = "highest",
-
                 nlTypes = "VIIRS.M",
-
                 nlPeriods = nlRange("201401", "201412"),
-
                 nlStats = list("sum", na.rm=TRUE),
-
                 configNames = "vcmcfg",
-
                 extensions = "avg_rade9",
-
                 removeGasFlaresMethod = "VTM")
2021-03-03 11:39:04: NlRange autodetected nlType: VIIRS.M 2021-03-03 11:39:04: Downloading ctry poly: YEM 2021-03-03 11:39:04: Downloading ctry shpZip: YEM 2021-03-03 11:39:04: Downloading https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_YEM_shp.zip trying URL 'https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_YEM_shp.zip ' Content type 'application/zip' length 1332199 bytes (1.3 MB) downloaded 1.3 MB

OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_0" with 1 features It has 2 fields 2021-03-03 11:39:09: Creating integer zone attribute col for polygon 2021-03-03 11:39:09: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_1" with 21 features It has 10 fields 2021-03-03 11:39:10: Creating integer zone attribute col for polygon 2021-03-03 11:39:10: Writing layer with new idx col Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_2" with 333 features It has 13 fields 2021-03-03 11:39:11: Creating integer zone attribute col for polygon 2021-03-03 11:39:11: Writing layer with new idx col Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. 2021-03-03 11:39:11: Saving shapefile as RDS for faster access 2021-03-03 11:39:11: Getting admLevels in YEM 2021-03-03 11:39:11: Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_0" with 1 features It has 3 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_1" with 21 features It has 11 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_2" with 333 features It has 14 fields 2021-03-03 11:39:13: Saving admLevel polygons as RDS 2021-03-03 11:39:13: Saving country admLevel structure to CSV 2021-03-03 11:39:13: ctryStruct not found. Creating Processing missing data: YEM:VIIRS.M:VCMCFG:avg_rade9:201401:sum, VIIRS.M:VCMCFG:avg_rade9:201402:sum, VIIRS.M:VCMCFG:avg_rade9:201403:sum, VIIRS.M:VCMCFG:avg_rade9:201404:sum, VIIRS.M:VCMCFG:avg_rade9:201405:sum, VIIRS.M:VCMCFG:avg_rade9:201406:sum, VIIRS.M:VCMCFG:avg_rade9:201407:sum, VIIRS.M:VCMCFG:avg_rade9:201408:sum, VIIRS.M:VCMCFG:avg_rade9:201409:sum, VIIRS.M:VCMCFG:avg_rade9:201410:sum, VIIRS.M:VCMCFG:avg_rade9:201411:sum, VIIRS.M:VCMCFG:avg_rade9:201412:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-03-03 11:39:14: START PROCESSING: ctryCodes=YEM, admLevels=list(YEM = "gadm36_YEM_1"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2021-03-03 11:39:14: Downloading country polygons ... | | 0%2021-03-03 11:39:14: Downloading polygon: YEM 2021-03-03 11:39:14: Downloading ctry poly: YEM 2021-03-03 11:39:14: Downloading ctry shpZip: YEM 2021-03-03 11:39:14: Polygon dir for YEM:3.6 already exists |=================================================================| 100% 2021-03-03 11:39:14: Downloading country polygons ... DONE 2021-03-03 11:39:14: PROCESSING nlType:VIIRS.M | configName: VCMCFG | extension: avg_rade9 | nlPeriod:201401 2021-03-03 11:39:14: Checking tiles required for VIIRS.M 201401 2021-03-03 11:39:14: YEM: Stats missing. Adding tiles 2021-03-03 11:39:14: numTiles: 1, Required tiles: 75N060W 2021-03-03 11:39:14: Downloading tile (1/1): 2014012 2021-03-03 11:39:14: Tif file exists, set Overwrite = TRUE to overwrite Error in message(Sys.time(), ": Extracting ", ntLtsZipLocalNamePathVIIRS) : object 'ntLtsZipLocalNamePathVIIRS' not found

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/chrisvwn/Rnightlights/issues/72#issuecomment-789488148, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQXQOWCZRTU3SLGLR3SGCLTBXNGNANCNFSM4WQS2MGQ .

chrisvwn commented 3 years ago

Confirmed. The github version needs to be updated. I will do so as soon as I can regain access.

On Thu, Mar 4, 2021 at 8:11 AM Chris Njuguna chris.njuguna@gmail.com wrote:

Hi,

I am not sure what the problem is. It could be that I am using a later version of the package as I have made some changes locally. Unfortunately, I am having trouble logging into my github account so I cannot update the code right away.

I will try to use the package as it is on github and see if I get the same errors.

On Wed, Mar 3, 2021 at 10:01 AM sxz745 notifications@github.com wrote:
Yes. The Session which produces persistent Error in availUrls[[nlType]][[configName]] : subscript out of bounds has the following session info.

sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale: [1] LC_COLLATE=English_United Kingdom.1252 [2] LC_CTYPE=English_United Kingdom.1252 [3] LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C [5] LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] reshape2_1.4.4 lubridate_1.7.10 Rnightlights_0.3.0.0

loaded via a namespace (and not attached): [1] httr_1.4.2 pkgload_1.2.0 maps_3.3.0 [4] jsonlite_1.7.2 foreach_1.5.1 dotCall64_1.0-1 [7] assertthat_0.2.1 sp_1.4-5 rworldmap_1.3-6 [10] selectr_0.4-2 remotes_2.2.0 sessioninfo_1.1.1 [13] pillar_1.5.0 lattice_0.20-38 glue_1.4.2 [16] rvest_0.3.6 plyr_1.8.6 pkgconfig_2.0.3 [19] devtools_2.3.2 raster_3.4-5 purrr_0.3.4 [22] fasterize_1.0.3 processx_3.4.5 ff_4.0.4 [25] tibble_3.1.0 generics_0.1.0 usethis_2.0.1 [28] ellipsis_0.3.1 cachem_1.0.4 withr_2.4.1 [31] cli_2.3.1 magrittr_2.0.1 crayon_1.4.1 [34] memoise_2.0.0 maptools_1.0-2 settings_0.2.4 [37] ps_1.5.0 fs_1.5.0 fansi_0.4.2 [40] cleangeo_0.2-3 xml2_1.3.2 foreign_0.8-75 [43] class_7.3-15 pkgbuild_1.2.0 tools_3.6.3 [46] data.table_1.14.0 prettyunits_1.1.1 hms_1.0.0 [49] lifecycle_1.0.0 stringr_1.4.0 callr_3.5.1 [52] compiler_3.6.3 e1071_1.7-4 rlang_0.4.10 [55] classInt_0.4-3 units_0.7-0 grid_3.6.3 [58] RCurl_1.98-1.2 iterators_1.0.13 spam_2.6-0 [61] bitops_1.0-6 tcltk_3.6.3 testthat_3.0.2 [64] codetools_0.2-16 DBI_1.1.1 curl_4.3 [67] R6_2.5.0 dplyr_1.0.4 rgdal_1.5-23 [70] fastmap_1.1.0 rgeos_0.5-5 bit_4.0.4 [73] utf8_1.1.4 rprojroot_2.0.2 KernSmooth_2.23-16 [76] readr_1.4.0 desc_1.2.0 stringi_1.5.3 [79] Rcpp_1.0.6 fields_11.6 vctrs_0.3.6 [82] sf_0.9-7 tidyselect_1.1.0

I can't really tell what went wrong. I tried the code for another country, e.g., YEM, but it still did't work and stopped with a different error message.

ctry <- getCtryNlData(ctryCode = "YEM",

-
                 admLevel = "highest",
-
                 nlTypes = "VIIRS.M",
-
                 nlPeriods = nlRange("201401", "201412"),
-
                 nlStats = list("sum", na.rm=TRUE),
-
                 configNames = "vcmcfg",
-
                 extensions = "avg_rade9",
-
                 removeGasFlaresMethod = "VTM")
2021-03-03 11:39:04: NlRange autodetected nlType: VIIRS.M 2021-03-03 11:39:04: Downloading ctry poly: YEM 2021-03-03 11:39:04: Downloading ctry shpZip: YEM 2021-03-03 11:39:04: Downloading https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_YEM_shp.zip trying URL ' https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_YEM_shp.zip' Content type 'application/zip' length 1332199 bytes (1.3 MB) downloaded 1.3 MB

OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_0" with 1 features It has 2 fields 2021-03-03 11:39:09: Creating integer zone attribute col for polygon 2021-03-03 11:39:09: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_1" with 21 features It has 10 fields 2021-03-03 11:39:10: Creating integer zone attribute col for polygon 2021-03-03 11:39:10: Writing layer with new idx col Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_2" with 333 features It has 13 fields 2021-03-03 11:39:11: Creating integer zone attribute col for polygon 2021-03-03 11:39:11: Writing layer with new idx col Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. 2021-03-03 11:39:11: Saving shapefile as RDS for faster access 2021-03-03 11:39:11: Getting admLevels in YEM 2021-03-03 11:39:11: Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_0" with 1 features It has 3 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_1" with 21 features It has 11 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\dell\AppData\Local\Temp\RtmpYH2CKy.Rnightlights\polygons\POLY_YEM_GADM-3.6-SHPZIP", layer: "gadm36_YEM_2" with 333 features It has 14 fields 2021-03-03 11:39:13: Saving admLevel polygons as RDS 2021-03-03 11:39:13: Saving country admLevel structure to CSV 2021-03-03 11:39:13: ctryStruct not found. Creating Processing missing data: YEM:VIIRS.M:VCMCFG:avg_rade9:201401:sum, VIIRS.M:VCMCFG:avg_rade9:201402:sum, VIIRS.M:VCMCFG:avg_rade9:201403:sum, VIIRS.M:VCMCFG:avg_rade9:201404:sum, VIIRS.M:VCMCFG:avg_rade9:201405:sum, VIIRS.M:VCMCFG:avg_rade9:201406:sum, VIIRS.M:VCMCFG:avg_rade9:201407:sum, VIIRS.M:VCMCFG:avg_rade9:201408:sum, VIIRS.M:VCMCFG:avg_rade9:201409:sum, VIIRS.M:VCMCFG:avg_rade9:201410:sum, VIIRS.M:VCMCFG:avg_rade9:201411:sum, VIIRS.M:VCMCFG:avg_rade9:201412:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-03-03 11:39:14: START PROCESSING: ctryCodes=YEM, admLevels=list(YEM = "gadm36_YEM_1"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("201401", "201402", "201403", "201404", "201405", "201406", "201407", "201408", "201409", "201410", "201411", "201412"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2021-03-03 11:39:14: Downloading country polygons ... 0%2021-03-03 11:39:14: Downloading polygon: YEM 2021-03-03 11:39:14: Downloading ctry poly: YEM 2021-03-03 11:39:14: Downloading ctry shpZip: YEM 2021-03-03 11:39:14: Polygon dir for YEM:3.6 already exists ================================================================= 100% 2021-03-03 11:39:14: Downloading country polygons ... DONE 2021-03-03 11:39:14: **** PROCESSING nlType:VIIRS.M configName: VCMCFG extension: avg_rade9 nlPeriod:201401**** 2021-03-03 11:39:14: Checking tiles required for VIIRS.M 201401 2021-03-03 11:39:14: YEM: Stats missing. Adding tiles 2021-03-03 11:39:14: numTiles: 1, Required tiles: 75N060W 2021-03-03 11:39:14: Downloading tile (1/1): 2014012 2021-03-03 11:39:14: Tif file exists, set Overwrite = TRUE to overwrite Error in message(Sys.time(), ": Extracting ", ntLtsZipLocalNamePathVIIRS)

object 'ntLtsZipLocalNamePathVIIRS' not found

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/chrisvwn/Rnightlights/issues/72#issuecomment-789488148, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQXQOWCZRTU3SLGLR3SGCLTBXNGNANCNFSM4WQS2MGQ .

chrisvwn commented 3 years ago

Hi @sxz745, I have updated the repo. Please update the package and try again.

sxz745 commented 3 years ago

Hi Chris, sorry to bother you again. Now I have installation errors. I've tried on two machines with R 3.6.3 and R 4.0.0 and encounter the same error:

devtools::install_github("chrisvwn/Rnightlights")

Downloading GitHub repo chrisvwn/Rnightlights@HEAD √ checking for file 'C:\Users\dell\AppData\Local\Temp\RtmpEFstxj\remotes12e0147a2fdc\chrisvwn-Rnightlights-f60f8e5/DESCRIPTION' ...

preparing 'Rnightlights': (914ms) √ checking DESCRIPTION meta-information ...
checking for LF line-endings in source and make files and shell scripts (775ms)
checking for empty or unneeded directories
building 'Rnightlights_0.3.0.0.tar.gz'
installing source package 'Rnightlights' ... using staged installation R Error in parse(outFile) : C:/Users/dell/AppData/Local/Temp/RtmpSEvRDe/R.INSTALL2d3c34974e3/Rnightlights/R/tiles.R:428:58: unexpected ')' 427: #' @export 428: getRadNlConfigExtension <- function(nlType, configName = ) ^ ERROR: unable to collate and parse R files for package 'Rnightlights'
removing 'C:/Program Files/R/R-3.6.3/library/Rnightlights'
restoring previous 'C:/Program Files/R/R-3.6.3/library/Rnightlights' Error: Failed to install 'Rnightlights' from GitHub: (converted from warning) installation of package ‘C:/Users/dell/AppData/Local/Temp/RtmpEFstxj/file12e03d645c5/Rnightlights_0.3.0.0.tar.gz’ had non-zero exit status

chrisvwn commented 3 years ago

I have fixed this. I uploaded while halfway editing a function. Apologies. Please try again.

chrisvwn commented 3 years ago

Closing this issue. Please reopen if you are still encountering it.