search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

custPolyPath Issue #46

Closed calvinke closed 4 years ago

calvinke commented 4 years ago

Hi,

I'm trying to extract the average night light for a series of circular zones within a country. I made an ESRI shape file, put the files into a zip folder, and added it to the directory. I tried to run this code below:

ctry <- "IND" getnldata_VIIRS<-getCtryNlData(custPolyPath="India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=TRUE)

However, I'm getting this error: "Error in admLevels[[cCodeIdx]] : subscript out of bounds". Can you help me with this problem please?

chrisvwn commented 4 years ago

Hi @calvinke ,

The custPolyPath feature is still very buggy. It does a best effort to discover the structure of the shapefile data, and this seems to be failing in your case. Could you send me a sample shapefile that I can have a look at?

calvinke commented 4 years ago

Here is a sample shapefile: link deleted Thanks for your help!

chrisvwn commented 4 years ago

Since the package was centered around GADM originally the ctryCode is still required by some functions. With time and more use we should be able to complete the separation.

For now, please supply the ctryCode parameter. You can specify any ctryCode so in your instance you can use ctryCode="IND".

Also:

Please try this:

ctry <- "IND"
getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)

If you have any issues do let me know. You may need to install the github version of the package for the latest updates.

calvinke commented 4 years ago

Thanks a lot. I tried doing that, and get this error: Error in rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode, : Cannot open data source When I searched that problem I found this page, is there a mismatch somewhere? https://github.com/chrisvwn/Rnightlights/blob/master/tests/testthat/test-admlevels.R

chrisvwn commented 4 years ago

Which version of Rnightlights do you have? The github version may be different if you had already installed from CRAN.

Alternatively, if you have the github version please reinstall because I made a few changes yesterday.

calvinke commented 4 years ago

I just installed the newest version Rnightlights_0.2.4.5 from github, but I got the same error:

getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)
2019-11-30 21:26:59: NlRange autodetected nlType: VIIRS.M
Error in rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode,  : 
  Cannot open data source
chrisvwn commented 4 years ago

Could you try putting the full path to "India.zip"?

On Sun, Dec 1, 2019, 05:29 calvinke notifications@github.com wrote:

I just installed the newest version Rnightlights_0.2.4.5 from github, but I got the same error:

getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE) 2019-11-30 21:26:59: NlRange autodetected nlType: VIIRS.M Error in rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode, : Cannot open data source

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/chrisvwn/Rnightlights/issues/46?email_source=notifications&email_token=ABQXQORZ5V7FGUBG4SKROULQWMOO7A5CNFSM4JSBGIW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFQZELY#issuecomment-560042543, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQXQORSBQ45PZSHHJ7PZILQWMOO7ANCNFSM4JSBGIWQ .

calvinke commented 4 years ago

I just tried that, same response

ctry <- "IND" getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C:\\Users\\Documents\\Data\\alan\\India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)

2019-12-01 12:37:43: NlRange autodetected nlType: VIIRS.M Error in rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode, : Cannot open data source

chrisvwn commented 4 years ago

Okay. Please delete anything referring to India.zip in your polygon folder. That is under the folder returned by getNlDataPath()

On Sun, Dec 1, 2019, 20:38 calvinke notifications@github.com wrote:

I just tried that, same response

ctry <- "IND" getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C:\Users\Documents\Data\alan\India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)

2019-12-01 12:37:43: NlRange autodetected nlType: VIIRS.M Error in rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode, : Cannot open data source

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/chrisvwn/Rnightlights/issues/46?email_source=notifications&email_token=ABQXQOUS4L5W3L4ZUVIWXKDQWPZBLA5CNFSM4JSBGIW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFRPZRA#issuecomment-560135364, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQXQOXVDCJFIJMW6MSPLV3QWPZBLANCNFSM4JSBGIWQ .

calvinke commented 4 years ago

OK I get a different message now: `> ctry <- "IND"

getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C:\Users\Documents\Data\alan\India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)`

2019-12-02 21:45:42: NlRange autodetected nlType: VIIRS.M 2019-12-02 21:45:42: Downloading ctry poly: IND 2019-12-02 21:45:42: Downloading ctry shpZip: IND 2019-12-02 21:45:42: Downloading file:///C:/Users/Documents/Data/alan/C:\Users\Documents\Data\alan\India.zip Error in utils::download.file(url = fullPolyUrl, destfile = getPolyFnameZip(ctryCode = ctryCode, : cannot open URL 'file:///C:/Users/Documents/Data/alan/C:\Users\Documents\Data\alan\India.zip', reason 'Invalid argument'

So I tried this code again:

> getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE) Result: 2019-12-02 21:53:38: NlRange autodetected nlType: VIIRS.M 2019-12-02 21:53:38: Downloading ctry poly: IND 2019-12-02 21:53:38: Downloading ctry shpZip: IND 2019-12-02 21:53:38: Downloading file:///C:/Users/Documents/Data/alan/India.zip Error in rgdal::ogrListLayers(getPolyFnamePath(ctryCode = ctryCode, gadmVersion = gadmVersion, : Cannot open data source In addition: Warning message: In utils::unzip(zipfile = polyFnameZip, junkpaths = TRUE, exdir = polyFnamePath) : error 1 in extracting from zip file

Sorry about all these questions, I am a beginner in R. Thanks for your help!

chrisvwn commented 4 years ago

No problem at all. This is not an easy task and the code is still under development. In fact, this helps me advance the work with custPolyPath :)

I have traced the error to a bug with handling Windows paths. I should have a quick fix in a few hours on github.

On Mon, Dec 2, 2019 at 3:55 PM calvinke notifications@github.com wrote:

OK I get a different message now: `> ctry <- "IND"

getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C:\Users\Documents\Data\alan\India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)`

2019-12-02 21:45:42: NlRange autodetected nlType: VIIRS.M 2019-12-02 21:45:42: Downloading ctry poly: IND 2019-12-02 21:45:42: Downloading ctry shpZip: IND 2019-12-02 21:45:42: Downloading file:///C:/Users/Documents/Data/alan/C:\Users\Documents\Data\alan\India.zip Error in utils::download.file(url = fullPolyUrl, destfile = getPolyFnameZip(ctryCode = ctryCode, : cannot open URL 'file:///C:/Users/Documents/Data/alan/C:\Users\Documents\Data\alan\India.zip', reason 'Invalid argument'

So I tried this code again:

getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE) Result: 2019-12-02 21:53:38: NlRange autodetected nlType: VIIRS.M 2019-12-02 21:53:38: Downloading ctry poly: IND 2019-12-02 21:53:38: Downloading ctry shpZip: IND 2019-12-02 21:53:38: Downloading file:///C:/Users/Documents/Data/alan/India.zip Error in rgdal::ogrListLayers(getPolyFnamePath(ctryCode = ctryCode, gadmVersion = gadmVersion, : Cannot open data source In addition: Warning message: In utils::unzip(zipfile = polyFnameZip, junkpaths = TRUE, exdir = polyFnamePath) : error 1 in extracting from zip file

Sorry about all these questions, I am a beginner in R. Thanks for your help!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/chrisvwn/Rnightlights/issues/46?email_source=notifications&email_token=ABQXQOQFSD7LDLX67MDSZK3QWUAT3A5CNFSM4JSBGIW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFTMTXY#issuecomment-560384479, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQXQOVUSSA5L7I6IZJA5ODQWUAT3ANCNFSM4JSBGIWQ .

chrisvwn commented 4 years ago

Okay. This is taking more than a few hours. I will update as soon as I can.

chrisvwn commented 4 years ago

Could you install the latest github version and try again. I think I have the issue resolved.

calvinke commented 4 years ago

Hi,

So doing the above codes gives me the same errors: 2019-12-05 22:22:39: NlRange autodetected nlType: VIIRS.M Error in rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode, : Cannot open data source However, based on the advice here https://github.com/r-spatial/sf/issues/673 , I tried to unzip the file into a folder and reference the SHP file. This is what happens: getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C:\\Users\\Documents\\Data\\alan\\India\\Buffer.shp", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE) 2019-12-05 22:24:46: NlRange autodetected nlType: VIIRS.M 2019-12-05 22:24:46: Downloading ctry poly: IND 2019-12-05 22:24:46: Downloading ctry shpZip: IND 2019-12-05 22:24:46: Downloading file:///C:\Users\Documents\Data\alan\India\Buffer.shp Error in utils::download.file(url = fullPolyUrl, destfile = getPolyFnameZip(ctryCode = ctryCode, : 'curl' call had nonzero exit status

chrisvwn commented 4 years ago

rgdal::ogrListLayers(path.expand(getPolyFnamePath(ctryCode = ctryCode, : Cannot open data source

This probably means the folder was created in the polygon directory but is empty. Could you try deleting any IND folders and files there and try again.

calvinke commented 4 years ago

Yes I just remembered you asked me to do that, this is what happens:

getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C:\\Users\\Documents\\Data\\alan\\India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE) 2019-12-05 22:35:42: NlRange autodetected nlType: VIIRS.M 2019-12-05 22:35:42: Downloading ctry poly: IND 2019-12-05 22:35:42: Downloading ctry shpZip: IND 2019-12-05 22:35:42: Downloading file:///C:\Users\Documents\Data\alan\India.zip Error in utils::download.file(url = fullPolyUrl, destfile = getPolyFnameZip(ctryCode = ctryCode, : 'curl' call had nonzero exit status

chrisvwn commented 4 years ago

Hmm. Do you have the curl library installed?

calvinke commented 4 years ago

Yes, I even reinstalled it and added library(curl) but same error.

chrisvwn commented 4 years ago

Could you try changing the backslashes to forward slashes? So instead of "C:\\Users\\Documents\\Data\\alan\\India.zip" try "C://Users/Documents/Data/alan/India.zip"

calvinke commented 4 years ago

I get the same problem:

ctry <- "IND" getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C://Users/Documents/Data/alan/India.zip", nlType="VIIRS.M", nlPeriods=nlRange("201501", "201612"), nlStats="sum", ignoreMissing=FALSE)

2019-12-06 13:13:43: NlRange autodetected nlType: VIIRS.M 2019-12-06 13:13:43: Downloading ctry poly: IND 2019-12-06 13:13:43: Downloading ctry shpZip: IND 2019-12-06 13:13:43: Downloading file:///C://Users/Documents/Data/alan/India.zip Error in utils::download.file(url = fullPolyUrl, destfile = getPolyFnameZip(ctryCode = ctryCode, : 'curl' call had nonzero exit status

chrisvwn commented 4 years ago

Which version of windows you have? You can find out by going to:

Start -> Run and type winver

It seems Windows versions before Windows 10 Build 1803 do not have curl natively installed. Thus, you may need to install curl separately.

calvinke commented 4 years ago

That did the trick, thanks! The issue now is that it has been processing for nearly 24 hours now. I'll try some of the performance improvements to see if that will make it run faster.

calvinke commented 4 years ago

Actually it looks like the code keeps looping:

`2019-12-07 15:58:02: NlRange autodetected nlType: VIIRS.M 2019-12-07 15:58:02: Downloading ctry poly: IND 2019-12-07 15:58:02: Downloading ctry shpZip: IND 2019-12-07 15:58:02: Downloading file:///C://Users/Documents/Data/alan/India.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 46.7M 100 46.7M 0 0 85.1M 0 --:--:-- --:--:-- --:--:-- 85.1M OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "IAGE71FL_Buffer" with 28526 features It has 24 fields Integer64 fields read as strings: ORIG_FID 2019-12-07 15:59:48: Creating integer zone attribute col for polygon 2019-12-07 16:00:05: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "IAGE71FL_Buffer" with 28526 features It has 25 fields 2019-12-07 16:01:40: Saving shapefile as RDS for faster access 2019-12-07 16:01:40: Getting admLevels in IND 2019-12-07 16:01:40: Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "0_IAGE71FL_Buffer" with 28526 features It has 25 fields 2019-12-07 16:02:55: Saving admLevel polygons as RDS 2019-12-07 16:03:15: Saving country admLevel structure to CSV 2019-12-07 16:03:15: ctryStruct not found. Creating OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "0_IAGE71FL_Buffer" with 28526 features It has 25 fields Processing missing data: IND:VIIRS.M:VCMCFG:201501:sum, VIIRS.M:VCMCFG:201502:sum, VIIRS.M:VCMCFG:201503:sum, VIIRS.M:VCMCFG:201504:sum, VIIRS.M:VCMCFG:201505:sum, VIIRS.M:VCMCFG:201506:sum, VIIRS.M:VCMCFG:201507:sum, VIIRS.M:VCMCFG:201508:sum, VIIRS.M:VCMCFG:201509:sum, VIIRS.M:VCMCFG:201510:sum, VIIRS.M:VCMCFG:201511:sum, VIIRS.M:VCMCFG:201512:sum, VIIRS.M:VCMCFG:201601:sum, VIIRS.M:VCMCFG:201602:sum, VIIRS.M:VCMCFG:201603:sum, VIIRS.M:VCMCFG:201604:sum, VIIRS.M:VCMCFG:201605:sum, VIIRS.M:VCMCFG:201606:sum, VIIRS.M:VCMCFG:201607:sum, VIIRS.M:VCMCFG:201608:sum, VIIRS.M:VCMCFG:201609:sum, VIIRS.M:VCMCFG:201610:sum, VIIRS.M:VCMCFG:201611:sum, VIIRS.M:VCMCFG:201612:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2019-12-07 16:05:13: START PROCESSING: ctryCodes=IND, admLevels=list(India.zip = "0_IAGE71FL_Buffer"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512", "201601", "201602", "201603", "201604", "201605", "201606", "201607", "201608", "201609", "201610", "201611", "201612"), nlStats=sum, custPolyPath=C://Users/Documents/Data/alan/India.zip, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2019-12-07 16:05:13: Downloading country polygons ... | | 0%2019-12-07 16:05:13: Downloading polygon: IND 2019-12-07 16:05:13: Downloading ctry poly: IND 2019-12-07 16:05:13: Downloading ctry shpZip: IND 2019-12-07 16:05:13: Polygon dir for IND:India.zip already exists |============================================================================================================| 100% 2019-12-07 16:05:13: Downloading country polygons ... DONE 2019-12-07 16:05:13: PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201501 2019-12-07 16:05:13: Checking tiles required for VIIRS.M 201501 2019-12-07 16:05:13: IND: Stats missing. Adding tiles 2019-12-07 16:05:16: numTiles: 1, Required tiles: 75N060E 2019-12-07 16:05:16: Downloading tile: 2015013 trying URL 'https://eogdata.mines.edu/download_dnb_composites_iframe.html' Content type 'text/html; charset=UTF-8' length 234051 bytes (228 KB) downloaded 228 KB

trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201501/vcmcfg/SVDNB_npp_20150101-20150131_75N060E_vcmcfg_v10_c201505111709.tgz' Content type 'application/x-gzip' length 596455258 bytes (568.8 MB) downloaded 568.8 MB

2019-12-07 16:08:34: Extracting C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201501_75N060E.tgz 2019-12-07 16:08:34: Getting list of files in C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201501_75N060E.tgz 2019-12-07 16:10:22: Extracting SVDNB_npp_20150101-20150131_75N060E_vcmcfg_v10_c201505111709.avg_rade9h.tif 2019-12-07 16:14:59: Processing ctryCode: IND 2019-12-07 16:14:59: Processing ctryCode: IND AdmLevel: 0_IAGE71FL_Buffer 2019-12-07 16:14:59: ProcessNlCountry: ctryCode=IND, admLevel=0_IAGE71FL_Buffer, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201501, nlStats=sum, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=C://Users/Documents/Data/alan/India.zip**** 2019-12-07 16:14:59: Check for existing data file 2019-12-07 16:14:59: Data file not found. Creating ... OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "0_IAGE71FL_Buffer" with 28526 features It has 25 fields 2019-12-07 16:17:00: Data file not found. Creating ... DONE 2019-12-07 16:17:00: Begin processing 201501 2019-12-07 16:17:00: Country output raster not found. Creating 2019-12-07 16:17:00: Load polygon layer for crop 2019-12-07 16:17:03: Reading in the raster tiles 2019-12-07 16:17:18: Cropping the raster tiles 2019-12-07 16:17:18: Gas flare removal is ON 2019-12-07 16:17:20: removeGasFlares = TRUE. Downloading gas flare country polygons. This will take a while trying URL 'https://www.ngdc.noaa.gov/eog/interest/gas_flares_countries_shapefiles.html' Content type 'text/html; charset=UTF-8' length unknown downloaded 13 KB

2019-12-07 16:17:22: Downloading gas flare polygon Flares_Algeria_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Algeria_1.tgz' Content type 'application/x-gzip' length 6265 bytes downloaded 6265 bytes

2019-12-07 16:17:23: Extracting Flares_Algeria_1.tgz 2019-12-07 16:17:23: Downloading gas flare polygon Flares_Angola_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Angola_1.tgz' Content type 'application/x-gzip' length 3334 bytes downloaded 3334 bytes

2019-12-07 16:17:23: Extracting Flares_Angola_1.tgz 2019-12-07 16:17:23: Downloading gas flare polygon Flares_Argentina_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Argentina_1.tgz' Content type 'application/x-gzip' length 10526 bytes (10 KB) downloaded 10 KB

2019-12-07 16:17:23: Extracting Flares_Argentina_1.tgz 2019-12-07 16:17:23: Downloading gas flare polygon Flares_Australia_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Australia_1.tgz' Content type 'application/x-gzip' length 5324 bytes downloaded 5324 bytes

2019-12-07 16:17:24: Extracting Flares_Australia_1.tgz 2019-12-07 16:17:24: Downloading gas flare polygon Flares_Azerbaijan_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Azerbaijan_1.tgz' Content type 'application/x-gzip' length 1885 bytes downloaded 1885 bytes

2019-12-07 16:17:24: Extracting Flares_Azerbaijan_1.tgz 2019-12-07 16:17:24: Downloading gas flare polygon Flares_Belgium_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Belgium_1.tgz' Content type 'application/x-gzip' length 1246 bytes downloaded 1246 bytes

2019-12-07 16:17:24: Extracting Flares_Belgium_1.tgz 2019-12-07 16:17:24: Downloading gas flare polygon Flares_Bolivia_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Bolivia_1.tgz' Content type 'application/x-gzip' length 2435 bytes downloaded 2435 bytes

2019-12-07 16:17:24: Extracting Flares_Bolivia_1.tgz 2019-12-07 16:17:24: Downloading gas flare polygon Flares_Brazil_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Brazil_1.tgz' Content type 'application/x-gzip' length 4375 bytes downloaded 4375 bytes

2019-12-07 16:17:25: Extracting Flares_Brazil_1.tgz 2019-12-07 16:17:25: Downloading gas flare polygon Flares_Brunei_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Brunei_1.tgz' Content type 'application/x-gzip' length 1909 bytes downloaded 1909 bytes

2019-12-07 16:17:25: Extracting Flares_Brunei_1.tgz 2019-12-07 16:17:25: Downloading gas flare polygon Flares_Cameroon_1.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_Cameroon_1.tgz' Content type 'application/x-gzip' length 1318 bytes downloaded 1318 bytes

... (redacted) ...

2019-12-07 16:17:46: Extracting Flares_Yemen_1.tgz 2019-12-07 16:17:46: Downloading gas flare polygon Flares_New_Zealand.tgz trying URL 'https://www.ngdc.noaa.gov/eog/data/web_data/gasflares_v2/country_vectors_20090618/Flares_New_Zealand.tgz' Content type 'application/x-gzip' length 1056 bytes downloaded 1056 bytes

2019-12-07 16:17:46: Extracting Flares_New_Zealand.tgz 2019-12-07 16:17:46: Creating global gas flare shapefile 2019-12-07 16:17:46: Reading in the downloaded gas flare polygons OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_Algeria_1", layer: "Flares_Algeria_1" with 11 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_Angola_1", layer: "Flares_Angola_1" with 4 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_Argentina_1", layer: "Flares_Argentina_1" with 18 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_Australia_1", layer: "Flares_Australia_1" with 10 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_Azerbaijan_1", layer: "Flares_Azerbaijan_1" with 3 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_Belgium_1", layer: "Flares_Belgium_1" with 1 features It has 1 fields

... (redacted)...

OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\gasflares\Flares_New_Zealand", layer: "Flares_New_Zealand" with 1 features It has 1 fields 2019-12-07 16:19:56: mosaicing 2019-12-07 16:19:56: cleaning the mosaiced polygon 2019-12-07 16:20:02: Saving mosaiced polygon to RDS 2019-12-07 16:20:06: Gas flare removal not required for: IND 2019-12-07 16:20:06: Cropping tile = 75N060E |============================================================================================================| 100%

2019-12-07 16:20:54: Masking the raster 2019-12-07 16:20:54: Mask using rasterize 2019-12-07 20:57:45: Writing the raster to disk 2019-12-07 20:58:05: Crop and mask using rasterize ... Done 2019-12-07 20:58:05: Begin extracting data from the raster |============================================================================================================| 100% 2019-12-07 21:18:29: Saving nlStat 'sum()' ... 2019-12-07 21:18:29: Saving nlStat 'sum' ... DONE 2019-12-07 21:18:29: DONE processing IND 201501 2019-12-07 21:18:29: COMPLETE. Writing data to disk 2019-12-07 21:18:29: ** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201502**** 2019-12-07 21:18:29: Checking tiles required for VIIRS.M 201502 2019-12-07 21:18:30: IND: Stats missing. Adding tiles 2019-12-07 21:18:30: numTiles: 1, Required tiles: 75N060E 2019-12-07 21:18:30: Downloading tile: 2015023 trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201502/vcmcfg/SVDNB_npp_20150201-20150228_75N060E_vcmcfg_v10_c201504281504.tgz' Content type 'application/x-gzip' length 554791758 bytes (529.1 MB) downloaded 529.1 MB

2019-12-07 21:22:00: Extracting C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201502_75N060E.tgz 2019-12-07 21:22:00: Getting list of files in C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201502_75N060E.tgz 2019-12-07 21:23:18: Extracting SVDNB_npp_20150201-20150228_75N060E_vcmcfg_v10_c201504281504.avg_rade9h.tif 2019-12-07 21:26:03: Processing ctryCode: IND 2019-12-07 21:26:03: Processing ctryCode: IND AdmLevel: 0_IAGE71FL_Buffer 2019-12-07 21:26:03: ProcessNlCountry: ctryCode=IND, admLevel=0_IAGE71FL_Buffer, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201502, nlStats=sum, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=C://Users/Documents/Data/DHS/alan/r/alan/India.zip**** 2019-12-07 21:26:03: Check for existing data file 2019-12-07 21:26:03: Data file found: C:/Users/Documents/.Rnightlights/data/NL_DATA_IND_IND_0_IAGE71FL_BUFFER_CUST-India.zip.csv 2019-12-07 21:26:03: Processing stats: sum() 2019-12-07 21:26:03: Load country data file 2019-12-07 21:26:03: Load country polygon admin level 2019-12-07 21:26:03: Begin processing 201502 2019-12-07 21:26:03: Country output raster not found. Creating 2019-12-07 21:26:03: Load polygon layer for crop 2019-12-07 21:26:05: Reading in the raster tiles 2019-12-07 21:26:17: Cropping the raster tiles 2019-12-07 21:26:17: Gas flare removal is ON 2019-12-07 21:26:22: Gas flare removal not required for: IND 2019-12-07 21:26:22: Cropping tile = 75N060E |============================================================================================================| 100%

2019-12-07 21:26:44: Masking the raster 2019-12-07 21:26:44: Mask using rasterize 2019-12-08 02:19:55: Writing the raster to disk 2019-12-08 02:20:15: Crop and mask using rasterize ... Done 2019-12-08 02:20:15: Begin extracting data from the raster |============================================================================================================| 100% 2019-12-08 02:40:42: Saving nlStat 'sum()' ... 2019-12-08 02:40:42: sum() already saved 2019-12-08 02:40:42: Saving nlStat 'sum' ... DONE 2019-12-08 02:40:42: DONE processing IND 201502 2019-12-08 02:40:42: COMPLETE. Writing data to disk 2019-12-08 02:40:43: ** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201503**** 2019-12-08 02:40:43: Checking tiles required for VIIRS.M 201503 2019-12-08 02:40:43: IND: Stats missing. Adding tiles 2019-12-08 02:40:43: numTiles: 1, Required tiles: 75N060E 2019-12-08 02:40:43: Downloading tile: 2015033 trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201503/vcmcfg/SVDNB_npp_20150301-20150331_75N060E_vcmcfg_v10_c201505191916.tgz' Content type 'application/x-gzip' length 464421920 bytes (442.9 MB) downloaded 442.9 MB

2019-12-08 02:43:07: Extracting C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201503_75N060E.tgz 2019-12-08 02:43:07: Getting list of files in C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201503_75N060E.tgz 2019-12-08 02:44:26: Extracting SVDNB_npp_20150301-20150331_75N060E_vcmcfg_v10_c201505191916.avg_rade9h.tif 2019-12-08 02:47:10: Processing ctryCode: IND 2019-12-08 02:47:10: Processing ctryCode: IND AdmLevel: 0_IAGE71FL_Buffer 2019-12-08 02:47:10: ProcessNlCountry: ctryCode=IND, admLevel=0_IAGE71FL_Buffer, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201503, nlStats=sum, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=C://Users/Documents/Data/alan/India.zip**** 2019-12-08 02:47:10: Check for existing data file 2019-12-08 02:47:10: Data file found: C:/Users/Documents/.Rnightlights/data/NL_DATA_IND_IND_0_IAGE71FL_BUFFER_CUST-India.zip.csv 2019-12-08 02:47:10: Processing stats: sum() 2019-12-08 02:47:10: Load country data file 2019-12-08 02:47:10: Load country polygon admin level 2019-12-08 02:47:10: Begin processing 201503 2019-12-08 02:47:10: Country output raster not found. Creating 2019-12-08 02:47:10: Load polygon layer for crop 2019-12-08 02:47:16: Reading in the raster tiles 2019-12-08 02:47:28: Cropping the raster tiles 2019-12-08 02:47:28: Gas flare removal is ON 2019-12-08 02:47:34: Gas flare removal not required for: IND 2019-12-08 02:47:34: Cropping tile = 75N060E |============================================================================================================| 100%

2019-12-08 02:47:55: Masking the raster 2019-12-08 02:47:55: Mask using rasterize 2019-12-08 07:19:44: Writing the raster to disk 2019-12-08 07:20:04: Crop and mask using rasterize ... Done 2019-12-08 07:20:04: Begin extracting data from the raster |============================================================================================================| 100% 2019-12-08 07:40:20: Saving nlStat 'sum()' ... 2019-12-08 07:40:20: sum() already saved 2019-12-08 07:40:20: Saving nlStat 'sum' ... DONE 2019-12-08 07:40:20: DONE processing IND 201503 2019-12-08 07:40:20: COMPLETE. Writing data to disk 2019-12-08 07:40:20: ** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201504**** 2019-12-08 07:40:20: Checking tiles required for VIIRS.M 201504 2019-12-08 07:40:20: IND: Stats missing. Adding tiles 2019-12-08 07:40:21: numTiles: 1, Required tiles: 75N060E 2019-12-08 07:40:21: Downloading tile: 2015043 trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201504/vcmcfg/SVDNB_npp_20150401-20150430_75N060E_vcmcfg_v10_c201506011707.tgz' Content type 'application/x-gzip' length 373663550 bytes (356.4 MB) downloaded 356.4 MB

2019-12-08 07:42:19: Extracting C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201504_75N060E.tgz 2019-12-08 07:42:19: Getting list of files in C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201504_75N060E.tgz 2019-12-08 07:43:33: Extracting SVDNB_npp_20150401-20150430_75N060E_vcmcfg_v10_c201506011707.avg_rade9h.tif 2019-12-08 07:46:14: Processing ctryCode: IND 2019-12-08 07:46:14: Processing ctryCode: IND AdmLevel: 0_IAGE71FL_Buffer 2019-12-08 07:46:14: ProcessNlCountry: ctryCode=IND, admLevel=0_IAGE71FL_Buffer, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201504, nlStats=sum, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=C://Users/Documents/Data/alan/India.zip**** 2019-12-08 07:46:14: Check for existing data file 2019-12-08 07:46:14: Data file found: C:/Users/Documents/.Rnightlights/data/NL_DATA_IND_IND_0_IAGE71FL_BUFFER_CUST-India.zip.csv 2019-12-08 07:46:14: Processing stats: sum() 2019-12-08 07:46:14: Load country data file 2019-12-08 07:46:15: Load country polygon admin level 2019-12-08 07:46:15: Begin processing 201504 2019-12-08 07:46:15: Country output raster not found. Creating 2019-12-08 07:46:15: Load polygon layer for crop 2019-12-08 07:46:17: Reading in the raster tiles 2019-12-08 07:46:29: Cropping the raster tiles 2019-12-08 07:46:29: Gas flare removal is ON 2019-12-08 07:46:35: Gas flare removal not required for: IND 2019-12-08 07:46:35: Cropping tile = 75N060E |============================================================================================================| 100%

2019-12-08 07:47:00: Masking the raster 2019-12-08 07:47:00: Mask using rasterize 2019-12-08 12:19:10: Writing the raster to disk 2019-12-08 12:19:32: Crop and mask using rasterize ... Done 2019-12-08 12:19:32: Begin extracting data from the raster |============================================================================================================| 100% 2019-12-08 12:39:56: Saving nlStat 'sum()' ... 2019-12-08 12:39:56: sum() already saved 2019-12-08 12:39:56: Saving nlStat 'sum' ... DONE 2019-12-08 12:39:56: DONE processing IND 201504 2019-12-08 12:39:56: COMPLETE. Writing data to disk 2019-12-08 12:39:57: ** PROCESSING nlType:VIIRS.M | configName: VCMCFG | nlPeriod:201505**** 2019-12-08 12:39:57: Checking tiles required for VIIRS.M 201505 2019-12-08 12:39:57: IND: Stats missing. Adding tiles 2019-12-08 12:39:57: numTiles: 1, Required tiles: 75N060E 2019-12-08 12:39:57: Downloading tile: 2015053 trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201505/vcmcfg/SVDNB_npp_20150501-20150531_75N060E_vcmcfg_v10_c201506161325.tgz' Content type 'application/x-gzip' length 303003422 bytes (289.0 MB) downloaded 289.0 MB

2019-12-08 12:41:41: Extracting C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201505_75N060E.tgz 2019-12-08 12:41:41: Getting list of files in C:/Users/Documents/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201505_75N060E.tgz 2019-12-08 12:42:50: Extracting SVDNB_npp_20150501-20150531_75N060E_vcmcfg_v10_c201506161325.avg_rade9h.tif 2019-12-08 12:45:46: Processing ctryCode: IND 2019-12-08 12:45:46: Processing ctryCode: IND AdmLevel: 0_IAGE71FL_Buffer 2019-12-08 12:45:46: ProcessNlCountry: ctryCode=IND, admLevel=0_IAGE71FL_Buffer, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201505, nlStats=sum, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=C://Users/Documents/Data/alan/India.zip**** 2019-12-08 12:45:46: Check for existing data file 2019-12-08 12:45:46: Data file found: C:/Users/Documents/.Rnightlights/data/NL_DATA_IND_IND_0_IAGE71FL_BUFFER_CUST-India.zip.csv 2019-12-08 12:45:46: Processing stats: sum() 2019-12-08 12:45:46: Load country data file 2019-12-08 12:45:46: Load country polygon admin level 2019-12-08 12:45:46: Begin processing 201505 2019-12-08 12:45:46: Country output raster not found. Creating 2019-12-08 12:45:46: Load polygon layer for crop 2019-12-08 12:45:48: Reading in the raster tiles 2019-12-08 12:46:00: Cropping the raster tiles 2019-12-08 12:46:00: Gas flare removal is ON 2019-12-08 12:46:06: Gas flare removal not required for: IND 2019-12-08 12:46:06: Cropping tile = 75N060E |============================================================================================================| 100%

2019-12-08 12:46:28: Masking the raster 2019-12-08 12:46:28: Mask using rasterize`

chrisvwn commented 4 years ago

This is fine. This is the code repeating the process for each nlPeriod. You can confirm with the lines that have the nlPeriod mentioned e.g.

2019-12-08 12:39:57: Checking tiles required for VIIRS.M 201505

You can give the performance enhancements a go. One other method that is not shown on the page is to have multiple getCtryNlData() functions running in different R sessions probably processing different years in parallel.

calvinke commented 4 years ago

Got it, thanks!

I changed to VIIRS.Y as this is what I actually need, but I got another error similar to issue 28 - will comment on that thread instead...

chrisvwn commented 4 years ago

Great! I am glad to hear this worked out! Closing issue.