Closed KipchumbaE closed 4 years ago
Hi @KipchumbaE ,
This is an odd problem. It sounds like there is a problem with the SSL certificate of the download site. However, I am not getting this error so maybe it was a temporary issue. I suggest you try again.
I have been also getting the same error message after trying from different machines. Is it possible to disable the cert checking when the data is being downloaded. Sharing the full message below for reference which ends in SSL peer certificate or SSH remote key was not OK
. Thanks!
> library(Rnightlights)
> ind_data <- getCtryNlData(ctryCode = "IND", admLevel = "highest",nlType = "VIIRS.M", nlPeriods = nlRange("201501", "201501","VIIRS.M"), nlStats = list("sum",na.rm=TRUE),ignoreMissing=FALSE)
2020-06-06 09:57:21: Downloading ctry poly: IND
2020-06-06 09:57:21: Downloading ctry shpZip: IND
2020-06-06 09:57:21: Downloading https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_IND_shp.zip
trying URL 'https://biogeo.ucdavis.edu/data/gadm3.6/shp/gadm36_IND_shp.zip'
Content type 'application/zip' length 17520295 bytes (16.7 MB)
==================================================
downloaded 16.7 MB
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_0"
with 1 features
It has 2 fields
2020-06-06 09:57:24: Creating integer zone attribute col for polygon
2020-06-06 09:57:24: Writing layer with new idx col
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_1"
with 36 features
It has 10 fields
2020-06-06 09:57:25: Creating integer zone attribute col for polygon
2020-06-06 09:57:25: Writing layer with new idx col
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_2"
with 666 features
It has 13 fields
2020-06-06 09:57:26: Creating integer zone attribute col for polygon
2020-06-06 09:57:26: Writing layer with new idx col
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_3"
with 2340 features
It has 16 fields
2020-06-06 09:57:27: Creating integer zone attribute col for polygon
2020-06-06 09:57:27: Writing layer with new idx col
2020-06-06 09:57:27: Saving shapefile as RDS for faster access
2020-06-06 09:57:27: Getting admLevels in IND
2020-06-06 09:57:27: Reading in all admLevels
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_0"
with 1 features
It has 3 fields
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_1"
with 36 features
It has 11 fields
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_2"
with 666 features
It has 14 fields
OGR data source with driver: ESRI Shapefile
Source: "/tmp/NIGHTLIGHTS_DATA/.Rnightlights/polygons/POLY_IND_GADM-3.6-SHPZIP", layer: "gadm36_IND_3"
with 2340 features
It has 17 fields
2020-06-06 09:57:30: Saving admLevel polygons as RDS
2020-06-06 09:57:33: Saving country admLevel structure to CSV
2020-06-06 09:57:33: ctryStruct not found. Creating
Processing missing data: IND:VIIRS.M:VCMCFG:201501:sum. This may take a while.
Note: Set 'ignoreMissing=TRUE' to return only data found or
'ignoreMissing=NULL' to return NULL if not all the data is found
2020-06-06 09:57:36: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201501, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-06-06 09:57:36: Downloading country polygons ...
| | 0%2020-06-06 09:57:36: Downloading polygon: IND
2020-06-06 09:57:36: Downloading ctry poly: IND
2020-06-06 09:57:36: Downloading ctry shpZip: IND
2020-06-06 09:57:36: Polygon dir for IND:3.6 already exists
|=======================================================================================================================| 100%
2020-06-06 09:57:36: Downloading country polygons ... DONE
2020-06-06 09:57:36: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201501****
2020-06-06 09:57:36: Checking tiles required for VIIRS.M 201501
2020-06-06 09:57:36: IND: Stats missing. Adding tiles
2020-06-06 09:57:36: numTiles: 1, Required tiles: 75N060E
2020-06-06 09:57:36: Downloading tile: 2015013
trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, :
cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
In addition: Warning message:
In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, :
URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'SSL peer certificate or SSH remote key was not OK'
> ind_data <- getCtryNlData(ctryCode = "IND", admLevel = "highest",nlType = "VIIRS.M", nlPeriods = nlRange("201501", "201501","VIIRS.M"), nlStats = list("sum",na.rm=TRUE),ignoreMissing=FALSE)
Processing missing data: IND:VIIRS.M:VCMCFG:201501:sum. This may take a while.
Note: Set 'ignoreMissing=TRUE' to return only data found or
'ignoreMissing=NULL' to return NULL if not all the data is found
2020-06-06 09:59:40: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201501, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-06-06 09:59:40: Downloading country polygons ...
| | 0%2020-06-06 09:59:40: Downloading polygon: IND
2020-06-06 09:59:40: Downloading ctry poly: IND
2020-06-06 09:59:40: Downloading ctry shpZip: IND
2020-06-06 09:59:40: Polygon dir for IND:3.6 already exists
|=======================================================================================================================| 100%
2020-06-06 09:59:40: Downloading country polygons ... DONE
2020-06-06 09:59:40: **** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201501****
2020-06-06 09:59:40: Checking tiles required for VIIRS.M 201501
2020-06-06 09:59:40: IND: Stats missing. Adding tiles
2020-06-06 09:59:41: numTiles: 1, Required tiles: 75N060E
2020-06-06 09:59:41: Downloading tile: 2015013
trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, :
cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'
In addition: Warning message:
In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, :
URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'SSL peer certificate or SSH remote key was not OK'
Hi @KipchumbaE and @xbsd,
Please tell me a bit about your environment. Are you behind a proxy server?
It's the same environment I have actually been using for a over a month now. This stopped working ~ 2 weeks back. I also tried from another server to check whether it was due to the local machine and got the same errors.
Both the local machine and the server is Linux based. No proxy, directly connected to the internet .. . I can wget the files without any issues, for eg.,
wget https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//202001/vcmcfg/SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz
--2020-06-06 22:29:31-- https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//202001/vcmcfg/SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz
Resolving eogdata.mines.edu (eogdata.mines.edu)... 138.67.208.254
Connecting to eogdata.mines.edu (eogdata.mines.edu)|138.67.208.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 364765833 (348M) [application/x-gzip]
Saving to: ‘SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz’
SVDNB_npp_20200101-20200131_00 100%[====================================================>] 347.87M 60.1MB/s in 5.9s
2020-06-06 22:29:38 (58.9 MB/s) - ‘SVDNB_npp_20200101-20200131_00N060W_vcmcfg_v10_c202002111500.tgz’ saved [364765833/364765833]
Hi @KipchumbaE and @xbsd,
Please tell me a bit about your environment. Are you behind a proxy server?
Am working on local machine directory not on a server
Thanks @xbsd @KipchumbaE Let me check this out. I will try a fresh R installation to replicate the error.
It worked after updating libcurl (sudo apt-get install libcurl4-openssl-dev
) and re-installing Rnightlights. I think there has probably been a CA update and the system hadn't been updated for some time.
Thank you @xbsd for the update! It would have taken me a while to figure that out.
@KipchumbaE Could you give this a go and let me know if the problem gets fixed?
I have re-installed it, unfortunately I still get the same error Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' In addition: Warning message: In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'Peer certificate cannot be authenticated with given CA certificates'
@KipchumbaE What version of libcurl do you have?
I'd add that ... when I manually tested - changing download from auto to curl with appropriate changes to the flags (eg., -N -R to preserve timestamp) worked. It would download the file. However download.file is set to method="auto" in various places and at some point I opted to simply re-install and apt-update the system to save time.
utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, method = "auto", extra = " -N --timestamping --no-use-server-timestamps")
On Friday, June 26, 2020, 06:17:54 AM EDT, Chris Njuguna <notifications@github.com> wrote:
@KipchumbaE What version of libcurl do you have?
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
I'd add that ... when I manually tested - changing download from auto to curl with appropriate changes to the flags (eg., -N -R to preserve timestamp) worked. It would download the file. However download.file is set to method="auto" in various places and at some point I opted to simply re-install and apt-update the system to save time.
utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, method = "auto", extra = " -N --timestamping --no-use-server-timestamps")
On Friday, June 26, 2020, 06:17:54 AM EDT, Chris Njuguna notifications@github.com wrote: @KipchumbaE What version of libcurl do you have? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@chrisvwn I have curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1
@xbsd Thanks for the additional info!
Indeed, this was the thinking behind offering different download options. That should one of the methods fail you can simply switch the download method.
I believe all downloads use the pkgOptions('downloadMethod')
except for the page index download and the GADM polygons download (GADM does not seem to like the aria2/accelerated download method).
I will look into this to see if I can streamline the use of downloadMethods.
@KipchumbaE Actually the ca-certificates will not be updated by curl. My bad.
Could you either try updating the ca certificates manually. I believe you can run:
sudo update-ca-certificates
Or try updating the whole system which should pick up the new certs as well.
I'd add that ... when I manually tested - changing download from auto to curl with appropriate changes to the flags (eg., -N -R to preserve timestamp) worked. It would download the file. However download.file is set to method="auto" in various places and at some point I opted to simply re-install and apt-update the system to save time.
utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, method = "auto", extra = " -N --timestamping --no-use-server-timestamps")
On Friday, June 26, 2020, 06:17:54 AM EDT, Chris Njuguna notifications@github.com wrote: @KipchumbaE What version of libcurl do you have? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
Hey @xbsd . I just realized this is about the index page download. I had made it so that we do not need to download it multiple times, however, this can be a problem as you noticed above I think due to the timestamps which is probably why your solution works. I think I will simply make it download the page afresh each time it runs. To force a re-download in the meantime you can delete the html page in the temp directory in the Rnightlights data directory. You can also run nlCleanup()
which will also clear any other temp files that may be lingering.
@KipchumbaE if you still have a problem please let us know. Otherwise, I think we can close this issue.
Hi Crhis am trying to extract data using this code wardAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "gadm36_KEN_3", nlType = "VIIRS.M", nlPeriods = nlRange("201401", "201412","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE) But I get the following error: Error in utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : cannot open URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' In addition: Warning message: In utils::download.file(url = ntLtsIndexUrlVIIRS, destfile = ntLtsPageLocalName, : URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html': status was 'Peer certificate cannot be authenticated with given CA certificates' Any idea what is the issue?