search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Error reporducing the KEN getCtryNlData example (Version 0.2.3, on both Cran and Github versions) #33

Closed lucasmation closed 5 years ago

lucasmation commented 5 years ago

Congrats for a great package!

I am trying to reproduce the example in the readme in a Windows enviroment. It seems that after downloading I get an error: 

Error in makeMPIcluster(spec, ...) : 
  the `Rmpi' package is needed for MPI clusters.

(full execution output below). From what I understood from researching, MPI is an external package which is cumbersome to install. I just want to run in the simplest for. I don't want any performance gains that these external packages may bring, it if forces me to do too much installation and bug fixing. 

#install.packages(“Rnightlights”)
#install.packages("lubridate")
#install.packages("reshape2")
#install.packages("ggplot2")

> library(Rnightlights)
Warning message:
package ‘Rnightlights’ was built under R version 3.5.3 
> library(lubridate)

Attaching package: ‘lubridate’

The following object is masked from ‘package:base’:

    date

> library(reshape2)
Warning message:
package ‘reshape2’ was built under R version 3.5.3 
> ctry <- "KEN"
> ?getCtryNlData
> pkgOptions("nlStats")
[1] "sum"  "mean"
> highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, 
+                                       admLevel = "highest",
+                                       nlType = "VIIRS.M", 
+                                       nlPeriods = nlRange("201401", "201412","VIIRS.M"), 
+                                       nlStats = list("sum",na.rm=TRUE),
+                                       ignoreMissing=FALSE)
Processing missing data: KEN:VIIRS.M:201401:sum, VIIRS.M:201402:sum, VIIRS.M:201403:sum, VIIRS.M:201404:sum, VIIRS.M:201405:sum, VIIRS.M:201406:sum, VIIRS.M:201407:sum, VIIRS.M:201408:sum, VIIRS.M:201409:sum, VIIRS.M:201410:sum, VIIRS.M:201411:sum, VIIRS.M:201412:sum. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2019-05-10 09:06:51: Downloading country polygons ...
2019-05-10 09:06:51: Downloading polygon: KEN
2019-05-10 09:06:51: Downloading ctry poly: KEN
2019-05-10 09:06:51: Polygon dir for KEN:3.6 already exists
2019-05-10 09:06:51: Downloading country polygons ... DONE
2019-05-10 09:06:52: **** PROCESSING nlType:VIIRS.M nlPeriod:201401****
2019-05-10 09:06:52: Checking tiles required for VIIRS.M 201401
2019-05-10 09:06:52: KEN: Stats missing. Adding tiles
2019-05-10 09:06:532 Required tiles: 75N060W,00N060W
2019-05-10 09:06:53: Downloading tile: 2014012
trying URL 'https://www.ngdc.noaa.gov/eog/viirs/download_dnb_composites_iframe.html'
Content type 'text/html; charset=UTF-8' length unknown
downloaded 257 KB

trying URL 'https://data.ngdc.noaa.gov/instruments/remote-sensing/passive/spectrometers-radiometers/imaging/viirs/dnb_composites/v10//201401/vcmcfg/SVDNB_npp_20140101-20140131_75N060W_vcmcfg_v10_c201506171538.tgz'
Content type 'application/x-gzip' length 611262356 bytes (582.9 MB)
downloaded 582.9 MB

2019-05-10 09:09:33: Extracting C:/Users/lucas/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_201401_75N060W.tgz
2019-05-10 09:09:33: Getting list of files in C:/Users/lucas/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_201401_75N060W.tgz
2019-05-10 09:10:00: Decompressing SVDNB_npp_20140101-20140131_75N060W_vcmcfg_v10_c201506171538.avg_rade9h.tif
2019-05-10 09:11:08: Downloading tile: 2014015
trying URL 'https://data.ngdc.noaa.gov/instruments/remote-sensing/passive/spectrometers-radiometers/imaging/viirs/dnb_composites/v10//201401/vcmcfg/SVDNB_npp_20140101-20140131_00N060W_vcmcfg_v10_c201506171538.tgz'
Content type 'application/x-gzip' length 324844599 bytes (309.8 MB)
downloaded 309.8 MB

2019-05-10 09:12:32: Extracting C:/Users/lucas/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_201401_00N060W.tgz
2019-05-10 09:12:32: Getting list of files in C:/Users/lucas/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_201401_00N060W.tgz
2019-05-10 09:13:04: Decompressing SVDNB_npp_20140101-20140131_00N060W_vcmcfg_v10_c201506171538.avg_rade9h.tif
2019-05-10 09:14:36: **processNLCountry: KEN gadm36_KEN_1 VIIRS.M 201401
2019-05-10 09:14:36: Check for existing data file
2019-05-10 09:14:36: Data file not found. Creating ...
2019-05-10 09:14:39: Data file not found. Creating ... DONE
2019-05-10 09:14:39: Load polygon layer for crop
2019-05-10 09:14:40: Begin processing 201401
2019-05-10 09:14:40: Reading in the raster tiles 
2019-05-10 09:14:42: Cropping the raster tiles 
2019-05-10 09:14:43: Cropping the raster tiles 
2019-05-10 09:14:45: Masking the merged raster 
2019-05-10 09:14:45: Mask using rasterize 
2019-05-10 09:14:48: Writing the merged raster to disk 
2019-05-10 09:14:50: Crop and mask using rasterize ... Done
2019-05-10 09:14:50: Begin extracting the data from the merged raster 
Loading required namespace: Rmpi
Failed with error:  ‘.onLoad failed in loadNamespace() for 'Rmpi', details:
  call: inDL(x, as.logical(local), as.logical(now), ...)
  error: unable to load shared object 'C:/Users/lucas/Documents/R/win-library/3.5/Rmpi/libs/x64/Rmpi.dll':
  LoadLibrary failure:  The specified module could not be found.
’
Error in makeMPIcluster(spec, ...) : 
  the `Rmpi' package is needed for MPI clusters.
chrisvwn commented 5 years ago

Hi @lucasmation

This is strange as the package does not use Rmpi. Please post the output of sessionInfo().

lucasmation commented 5 years ago

Sure, tks in advance @chrisvwn , 

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] reshape2_1.4.3     lubridate_1.7.4    Rnightlights_0.2.3

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0        lattice_0.20-35   codetools_0.2-15 
 [4] cleangeo_0.2-2    foreach_1.4.4     plyr_1.8.4       
 [7] rworldmap_1.3-6   grid_3.5.1        magrittr_1.5     
[10] spam_2.2-2        stringi_1.2.4     settings_0.2.4   
[13] data.table_1.12.2 sp_1.3-1          rgdal_1.4-3      
[16] iterators_1.0.10  tools_3.5.1       stringr_1.4.0    
[19] foreign_0.8-70    maps_3.3.0        fields_9.7       
[22] compiler_3.5.1    maptools_0.9-5    dotCall64_1.0-0  
[25] rgeos_0.4-3
chrisvwn commented 5 years ago

Hmm. This all looks okay. Are you using the Rmpi package? Do you have settings for example in your .Renviron that could be causing the creation of MPI clusters rather than CPU clusters? Rnightlights should not be using it but it seems maybe something else is influencing it.

Rmpi also seems to not be properly installed as indicated by the error message about a missing dll. If possible maybe uninstall it (and remove any related configs in R) and try running again?

lucasmation commented 5 years ago

No. My .Renviron is empty. I never installed MPI. I installed the package Rmpi the first time Rnightlights gave me this error message.

chrisvwn commented 5 years ago

Could you check if you have anything in your Makevars config file?

chrisvwn commented 5 years ago

Actually, could you try checking the default cluster type in your environment by running:

snow::getClusterOption("type")

If this does not yield SOCK (I suspect you will have MPI) try setting the default back to 'SOCK' by running:

snow::setDefaultClusterOptions(type='SOCK')
chrisvwn commented 5 years ago

Hi @lucasmation. Were you able to try out the recommendations above?

chrisvwn commented 5 years ago

Closing this issue. Please reopen if you are still facing this.