search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Ward-level nightlight data #63

Closed KipchumbaE closed 4 years ago

KipchumbaE commented 4 years ago

Hi Crhis am trying to extract data using this code wardAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "gadm36_KEN_3", nlType = "VIIRS.M", nlPeriods = nlRange("201401", "201412","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE) But I get the following error: Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' In addition: Warning message: In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'Peer certificate cannot be authenticated with given CA certificates' Any idea what is the issue?

chrisvwn commented 4 years ago

Hi @KipchumbaE ,

This is an odd problem. It sounds like there is a problem with the SSL certificate of the download site. However, I am not getting this error so maybe it was a temporary issue. I suggest you try again.

xbsd commented 4 years ago

I have been also getting the same error message after trying from different machines. Is it possible to disable the cert checking when the data is being downloaded. Sharing the full message below for reference which ends in SSL peer certificate or SSH remote key was not OK. Thanks!

> library(Rnightlights)
> ind_data <- getCtryNlData(ctryCode = "IND", admLevel = "highest",nlType = "VIIRS.M", nlPeriods = nlRange("201501", "201501","VIIRS.M"), nlStats = list("sum",na.rm=TRUE),ignoreMissing=FALSE)

2020-06-06 09:57:21: Downloading ctry poly: IND
2020-06-06 09:57:21: Downloading ctry shpZip: IND
2020-06-06 09:57:21: Downloading https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_IND_shp.zip
trying URL 'https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_IND_shp.zip'
Content type 'application/zip' length 17520295 bytes (16.7 MB)
==================================================
downloaded 16.7 MB

OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_0"
with 1 features
It has 2 fields
2020-06-06 09:57:24: Creating integer zone attribute col for polygon
2020-06-06 09:57:24: Writing layer with new idx col
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_1"
with 36 features
It has 10 fields
2020-06-06 09:57:25: Creating integer zone attribute col for polygon
2020-06-06 09:57:25: Writing layer with new idx col
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_2"
with 666 features
It has 13 fields
2020-06-06 09:57:26: Creating integer zone attribute col for polygon
2020-06-06 09:57:26: Writing layer with new idx col
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_3"
with 2340 features
It has 16 fields
2020-06-06 09:57:27: Creating integer zone attribute col for polygon
2020-06-06 09:57:27: Writing layer with new idx col
2020-06-06 09:57:27: Saving shapefile as RDS for faster access
2020-06-06 09:57:27: Getting admLevels in IND
2020-06-06 09:57:27: Reading in all admLevels
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_0"
with 1 features
It has 3 fields
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_1"
with 36 features
It has 11 fields
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_2"
with 666 features
It has 14 fields
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_3"
with 2340 features
It has 17 fields
2020-06-06 09:57:30: Saving admLevel polygons as RDS
2020-06-06 09:57:33: Saving country admLevel structure to CSV
2020-06-06 09:57:33: ctryStruct not found. Creating
Processing missing data: IND:VIIRS.M:VCMCFG:201501:sum. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2020-06-06 09:57:36: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201501, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-06-06 09:57:36: Downloading country polygons ...
  |                                                                                                                       |   0%2020-06-06 09:57:36: Downloading polygon: IND
2020-06-06 09:57:36: Downloading ctry poly: IND
2020-06-06 09:57:36: Downloading ctry shpZip: IND
2020-06-06 09:57:36: Polygon dir for IND:3.6 already exists
  |=======================================================================================================================| 100%
2020-06-06 09:57:36: Downloading country polygons ... DONE
2020-06-06 09:57:36: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201501****
2020-06-06 09:57:36: Checking tiles required for VIIRS.M 201501
2020-06-06 09:57:36: IND: Stats missing. Adding tiles
2020-06-06 09:57:36: numTiles: 1, Required tiles: 75N060E
2020-06-06 09:57:36: Downloading tile: 2015013
trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName,  : 
  cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
In addition: Warning message:
In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName,  :
  URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'SSL peer certificate or SSH remote key was not OK'
> ind_data <- getCtryNlData(ctryCode = "IND", admLevel = "highest",nlType = "VIIRS.M", nlPeriods = nlRange("201501", "201501","VIIRS.M"), nlStats = list("sum",na.rm=TRUE),ignoreMissing=FALSE)
Processing missing data: IND:VIIRS.M:VCMCFG:201501:sum. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2020-06-06 09:59:40: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201501, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-06-06 09:59:40: Downloading country polygons ...
  |                                                                                                                       |   0%2020-06-06 09:59:40: Downloading polygon: IND
2020-06-06 09:59:40: Downloading ctry poly: IND
2020-06-06 09:59:40: Downloading ctry shpZip: IND
2020-06-06 09:59:40: Polygon dir for IND:3.6 already exists
  |=======================================================================================================================| 100%
2020-06-06 09:59:40: Downloading country polygons ... DONE
2020-06-06 09:59:40: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201501****
2020-06-06 09:59:40: Checking tiles required for VIIRS.M 201501
2020-06-06 09:59:40: IND: Stats missing. Adding tiles
2020-06-06 09:59:41: numTiles: 1, Required tiles: 75N060E
2020-06-06 09:59:41: Downloading tile: 2015013
trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName,  : 
  cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
In addition: Warning message:
In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName,  :
  URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'SSL peer certificate or SSH remote key was not OK'
chrisvwn commented 4 years ago

Hi @KipchumbaE and @xbsd,

Please tell me a bit about your environment. Are you behind a proxy server?

xbsd commented 4 years ago

It's the same environment I have actually been using for a over a month now. This stopped working ~ 2 weeks back. I also tried from another server to check whether it was due to the local machine and got the same errors.

Both the local machine and the server is Linux based. No proxy, directly connected to the internet .. . I can wget the files without any issues, for eg.,

wget https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//202001/vcmcfg/SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz
--2020-06-06 22:29:31--  https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//202001/vcmcfg/SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz
Resolving eogdata.mines.edu (eogdata.mines.edu)... 138.67.208.254
Connecting to eogdata.mines.edu (eogdata.mines.edu)|138.67.208.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 364765833 (348M) [application/x-gzip]
Saving to: ‘SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz’

SVDNB_npp_20200101-20200131_00 100%[====================================================>] 347.87M  60.1MB/s    in 5.9s

2020-06-06 22:29:38 (58.9 MB/s) - ‘SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz’ saved [364765833/364765833]
KipchumbaE commented 4 years ago

Hi @KipchumbaE and @xbsd,

Please tell me a bit about your environment. Are you behind a proxy server?

Am working on local machine directory not on a server

chrisvwn commented 4 years ago

Thanks @xbsd @KipchumbaE Let me check this out. I will try a fresh R installation to replicate the error.

xbsd commented 4 years ago

It worked after updating libcurl (sudo apt-get install libcurl4-openssl-dev) and re-installing Rnightlights. I think there has probably been a CA update and the system hadn't been updated for some time.

chrisvwn commented 4 years ago

Thank you @xbsd for the update! It would have taken me a while to figure that out.

@KipchumbaE Could you give this a go and let me know if the problem gets fixed?

KipchumbaE commented 4 years ago

I have re-installed it, unfortunately I still get the same error Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' In addition: Warning message: In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'Peer certificate cannot be authenticated with given CA certificates'

chrisvwn commented 4 years ago

@KipchumbaE What version of libcurl do you have?

xbsd commented 4 years ago

I'd add that ... when I manually tested - changing download from auto to curl with appropriate changes to the flags (eg., -N -R to preserve timestamp) worked. It would download the file. However download.file is set to method="auto" in various places and at some point I opted to simply re-install and apt-update the system to save time. utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, method = "auto", extra = " -N --timestamping --no-use-server-timestamps")

On Friday, June 26, 2020, 06:17:54 AM EDT, Chris Njuguna <notifications@github.com> wrote:

@KipchumbaE What version of libcurl do you have?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

KipchumbaE commented 4 years ago

I'd add that ... when I manually tested - changing download from auto to curl with appropriate changes to the flags (eg., -N -R to preserve timestamp) worked. It would download the file. However download.file is set to method="auto" in various places and at some point I opted to simply re-install and apt-update the system to save time. utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, method = "auto", extra = " -N --timestamping --no-use-server-timestamps") On Friday, June 26, 2020, 06:17:54 AM EDT, Chris Njuguna notifications@github.com wrote: @KipchumbaE What version of libcurl do you have? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

@chrisvwn I have curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1

chrisvwn commented 4 years ago

@xbsd Thanks for the additional info!

Indeed, this was the thinking behind offering different download options. That should one of the methods fail you can simply switch the download method.

I believe all downloads use the pkgOptions('downloadMethod') except for the page index download and the GADM polygons download (GADM does not seem to like the aria2/accelerated download method).

I will look into this to see if I can streamline the use of downloadMethods.

chrisvwn commented 4 years ago

@KipchumbaE Actually the ca-certificates will not be updated by curl. My bad.

Could you either try updating the ca certificates manually. I believe you can run:

sudo update-ca-certificates

Or try updating the whole system which should pick up the new certs as well.

chrisvwn commented 4 years ago

I'd add that ... when I manually tested - changing download from auto to curl with appropriate changes to the flags (eg., -N -R to preserve timestamp) worked. It would download the file. However download.file is set to method="auto" in various places and at some point I opted to simply re-install and apt-update the system to save time. utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, method = "auto", extra = " -N --timestamping --no-use-server-timestamps") On Friday, June 26, 2020, 06:17:54 AM EDT, Chris Njuguna notifications@github.com wrote: @KipchumbaE What version of libcurl do you have? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Hey @xbsd . I just realized this is about the index page download. I had made it so that we do not need to download it multiple times, however, this can be a problem as you noticed above I think due to the timestamps which is probably why your solution works. I think I will simply make it download the page afresh each time it runs. To force a re-download in the meantime you can delete the html page in the temp directory in the Rnightlights data directory. You can also run nlCleanup() which will also clear any other temp files that may be lingering.

chrisvwn commented 4 years ago

@KipchumbaE if you still have a problem please let us know. Otherwise, I think we can close this issue.