chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Failed to parse? #56

Closed HaasL closed 4 years ago

HaasL commented 4 years ago

Hey Chris, I downloaded the data but fail to complete the lasts steps. The download seems to be successful, but when trying to plot it says "all formats failed to parse." Can you tell me what went wrong ?

> ctry <- "MMR" #replace to run for any other country
> 
> #download and process monthly VIIRS stats at the highest admin level
> highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, 
+                                       #admLevel="gadm36_MMR_3",
+                                       admLevel = "highest",
+                                       nlType = "VIIRS.M", 
+                                       nlPeriods = nlRange("201501", "201512","VIIRS.M"), 
+                                       nlStats = list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)),
+                                       ignoreMissing=FALSE)
Processing missing data: MMR:VIIRS.M:VCMCFG:201510:sum, VIIRS.M:VCMCFG:201511:sum, VIIRS.M:VCMCFG:201512:sum, VIIRS.M:VCMCFG:201510:median, VIIRS.M:VCMCFG:201511:median, VIIRS.M:VCMCFG:201512:median. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2020-04-12 16:40:48: **** START PROCESSING: ctryCodes=MMR, admLevels=list(MMR = "gadm36_MMR_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:48: Downloading country polygons ...
2020-04-12 16:40:48: Downloading polygon: MMR
2020-04-12 16:40:48: Downloading ctry poly: MMR
2020-04-12 16:40:48: Downloading ctry shpZip: MMR
2020-04-12 16:40:48: Polygon dir for MMR:3.6 already exists
2020-04-12 16:40:48: Downloading country polygons ... DONE
2020-04-12 16:40:48: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201501****
2020-04-12 16:40:48: Checking tiles required for VIIRS.M 201501
2020-04-12 16:40:48: MMR: Stats missing. Adding tiles
2020-04-12 16:40:48: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:48: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201501_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:48: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201501, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:48: Check for existing data file
2020-04-12 16:40:48: Load polygon layer for crop
2020-04-12 16:40:48: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:48: **All stats exist for MMR gadm36_MMR_1 201501. Skipping
2020-04-12 16:40:48: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:48: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201502****
2020-04-12 16:40:48: Checking tiles required for VIIRS.M 201502
2020-04-12 16:40:48: MMR: Stats missing. Adding tiles
2020-04-12 16:40:49: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:49: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201502_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:49: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201502, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:49: Check for existing data file
2020-04-12 16:40:49: Load polygon layer for crop
2020-04-12 16:40:49: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:49: **All stats exist for MMR gadm36_MMR_1 201502. Skipping
2020-04-12 16:40:49: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:49: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201503****
2020-04-12 16:40:49: Checking tiles required for VIIRS.M 201503
2020-04-12 16:40:49: MMR: Stats missing. Adding tiles
2020-04-12 16:40:49: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:49: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201503_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:49: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201503, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:49: Check for existing data file
2020-04-12 16:40:49: Load polygon layer for crop
2020-04-12 16:40:49: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:49: **All stats exist for MMR gadm36_MMR_1 201503. Skipping
2020-04-12 16:40:49: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:49: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201504****
2020-04-12 16:40:49: Checking tiles required for VIIRS.M 201504
2020-04-12 16:40:49: MMR: Stats missing. Adding tiles
2020-04-12 16:40:49: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:49: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201504_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:49: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201504, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:49: Check for existing data file
2020-04-12 16:40:49: Load polygon layer for crop
2020-04-12 16:40:50: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:50: **All stats exist for MMR gadm36_MMR_1 201504. Skipping
2020-04-12 16:40:50: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:50: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201505****
2020-04-12 16:40:50: Checking tiles required for VIIRS.M 201505
2020-04-12 16:40:50: MMR: Stats missing. Adding tiles
2020-04-12 16:40:50: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:50: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201505_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:50: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201505, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:50: Check for existing data file
2020-04-12 16:40:50: Load polygon layer for crop
2020-04-12 16:40:50: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:50: **All stats exist for MMR gadm36_MMR_1 201505. Skipping
2020-04-12 16:40:50: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:50: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201506****
2020-04-12 16:40:50: Checking tiles required for VIIRS.M 201506
2020-04-12 16:40:50: MMR: Stats missing. Adding tiles
2020-04-12 16:40:50: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:50: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201506_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:50: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201506, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:50: Check for existing data file
2020-04-12 16:40:50: Load polygon layer for crop
2020-04-12 16:40:51: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:51: **All stats exist for MMR gadm36_MMR_1 201506. Skipping
2020-04-12 16:40:51: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:51: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201507****
2020-04-12 16:40:51: Checking tiles required for VIIRS.M 201507
2020-04-12 16:40:51: MMR: Stats missing. Adding tiles
2020-04-12 16:40:51: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:51: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201507_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:51: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201507, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:51: Check for existing data file
2020-04-12 16:40:51: Load polygon layer for crop
2020-04-12 16:40:51: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:51: **All stats exist for MMR gadm36_MMR_1 201507. Skipping
2020-04-12 16:40:51: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:51: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201508****
2020-04-12 16:40:51: Checking tiles required for VIIRS.M 201508
2020-04-12 16:40:51: MMR: Stats missing. Adding tiles
2020-04-12 16:40:51: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:51: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201508_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:51: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201508, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:51: Check for existing data file
2020-04-12 16:40:51: Load polygon layer for crop
2020-04-12 16:40:51: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:51: **All stats exist for MMR gadm36_MMR_1 201508. Skipping
2020-04-12 16:40:51: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:51: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201509****
2020-04-12 16:40:51: Checking tiles required for VIIRS.M 201509
2020-04-12 16:40:51: MMR: Stats missing. Adding tiles
2020-04-12 16:40:51: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:51: Cropped raster /Users/.Rnightlights/outputrasters/NL_MMR_VIIRS.M_201509_VCMCFG-MTSALL-MEAN-RGFT_GADM-3.6-SHPZIP.tif already exists. Skipping tile download
2020-04-12 16:40:51: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201509, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:40:51: Check for existing data file
2020-04-12 16:40:51: Load polygon layer for crop
2020-04-12 16:40:52: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:40:52: **All stats exist for MMR gadm36_MMR_1 201509. Skipping
2020-04-12 16:40:52: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:40:52: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201510****
2020-04-12 16:40:52: Checking tiles required for VIIRS.M 201510
2020-04-12 16:40:52: MMR: Stats missing. Adding tiles
2020-04-12 16:40:52: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:40:52: Downloading tile: 2015103
trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201510/vcmcfg/SVDNB_npp_20151001-20151031_75N060E_vcmcfg_v10_c201511181404.tgz'
Content type 'application/x-gzip' length 517747610 bytes (493.8 MB)
==================================================
downloaded 493.8 MB

2020-04-12 16:43:00: Extracting /Users/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201510_75N060E.tgz
2020-04-12 16:43:00: Getting list of files in /Users/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201510_75N060E.tgz
2020-04-12 16:43:43: Extracting SVDNB_npp_20151001-20151031_75N060E_vcmcfg_v10_c201511181404.avg_rade9h.tif
2020-04-12 16:45:03: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201510, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:45:03: Check for existing data file
2020-04-12 16:45:03: Load polygon layer for crop
2020-04-12 16:45:03: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:45:03: Processing stats: list("sum", na.rm = TRUE),list("median", na.rm = TRUE)
2020-04-12 16:45:03: Load country data file
2020-04-12 16:45:03: Load country polygon admin level
2020-04-12 16:45:03: Begin processing 201510
2020-04-12 16:45:03: Country output raster not found. Creating
2020-04-12 16:45:03: Reading in the raster tiles
2020-04-12 16:45:04: Cropping the raster tiles 
2020-04-12 16:45:04: Gas flare removal is ON
2020-04-12 16:45:06: Gas flare removal not required for: MMR
2020-04-12 16:45:06: Cropping tile = 75N060E
  |================================================================================================================================| 100%

2020-04-12 16:45:11: Masking the raster 
2020-04-12 16:45:11: Mask using rasterize 
2020-04-12 16:45:53: Writing the raster to disk 
  |================================================================================================================================| 100%

2020-04-12 16:45:56: Crop and mask using rasterize ... Done
2020-04-12 16:45:56: Begin extracting data from the raster 
  |================================================================================================================================| 100%
2020-04-12 16:46:08: DONE processing MMR 201510
2020-04-12 16:46:08: **COMPLETE. Writing data to disk
2020-04-12 16:46:08: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:46:08: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201511****
2020-04-12 16:46:08: Checking tiles required for VIIRS.M 201511
2020-04-12 16:46:08: MMR: Stats missing. Adding tiles
2020-04-12 16:46:08: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:46:08: Downloading tile: 2015113
trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201511/vcmcfg/SVDNB_npp_20151101-20151130_75N060E_vcmcfg_v10_c201512121648.tgz'
Content type 'application/x-gzip' length 605310334 bytes (577.3 MB)
==================================================
downloaded 577.3 MB

2020-04-12 16:48:31: Extracting /Users/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201511_75N060E.tgz
2020-04-12 16:48:31: Getting list of files in /Users/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201511_75N060E.tgz
2020-04-12 16:49:05: Extracting SVDNB_npp_20151101-20151130_75N060E_vcmcfg_v10_c201512121648.avg_rade9h.tif
2020-04-12 16:50:24: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201511, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:50:24: Check for existing data file
2020-04-12 16:50:24: Load polygon layer for crop
2020-04-12 16:50:25: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:50:25: Processing stats: list("sum", na.rm = TRUE),list("median", na.rm = TRUE)
2020-04-12 16:50:25: Load country data file
2020-04-12 16:50:25: Load country polygon admin level
2020-04-12 16:50:25: Begin processing 201511
2020-04-12 16:50:25: Country output raster not found. Creating
2020-04-12 16:50:25: Reading in the raster tiles
2020-04-12 16:50:25: Cropping the raster tiles 
2020-04-12 16:50:25: Gas flare removal is ON
2020-04-12 16:50:26: Gas flare removal not required for: MMR
2020-04-12 16:50:26: Cropping tile = 75N060E
  |================================================================================================================================| 100%

2020-04-12 16:50:31: Masking the raster 
2020-04-12 16:50:31: Mask using rasterize 
2020-04-12 16:51:10: Writing the raster to disk 
  |================================================================================================================================| 100%

2020-04-12 16:51:13: Crop and mask using rasterize ... Done
2020-04-12 16:51:13: Begin extracting data from the raster 
  |================================================================================================================================| 100%
2020-04-12 16:51:24: DONE processing MMR 201511
2020-04-12 16:51:24: **COMPLETE. Writing data to disk
2020-04-12 16:51:24: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:51:24: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201512****
2020-04-12 16:51:24: Checking tiles required for VIIRS.M 201512
2020-04-12 16:51:24: MMR: Stats missing. Adding tiles
2020-04-12 16:51:24: numTiles: 1, Required tiles: 75N060E
2020-04-12 16:51:24: Downloading tile: 2015123
trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201512/vcmcfg/SVDNB_npp_20151201-20151231_75N060E_vcmcfg_v10_c201601251413.tgz'
Content type 'application/x-gzip' length 619810462 bytes (591.1 MB)
==================================================
downloaded 591.1 MB

2020-04-12 16:53:50: Extracting /Users/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201512_75N060E.tgz
2020-04-12 16:53:50: Getting list of files in /Users/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201512_75N060E.tgz
2020-04-12 16:54:24: Extracting SVDNB_npp_20151201-20151231_75N060E_vcmcfg_v10_c201601251413.avg_rade9h.tif
2020-04-12 16:55:43: ProcessNlCountry: ctryCode=MMR, admLevel=gadm36_MMR_1, nlType=VIIRS.M, configName=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriod=201512, nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast, gadmVersion=3.6, gadmPolyType=shpZip, custPolyPath=NULL****
2020-04-12 16:55:43: Check for existing data file
2020-04-12 16:55:43: Load polygon layer for crop
2020-04-12 16:55:43: Data file found: /Users/.Rnightlights/data/NL_DATA_MMR_ADM1_GADM-3.6-SHPZIP.csv
2020-04-12 16:55:43: Processing stats: list("sum", na.rm = TRUE),list("median", na.rm = TRUE)
2020-04-12 16:55:43: Load country data file
2020-04-12 16:55:43: Load country polygon admin level
2020-04-12 16:55:43: Begin processing 201512
2020-04-12 16:55:43: Country output raster not found. Creating
2020-04-12 16:55:43: Reading in the raster tiles
2020-04-12 16:55:44: Cropping the raster tiles 
2020-04-12 16:55:44: Gas flare removal is ON
2020-04-12 16:55:45: Gas flare removal not required for: MMR
2020-04-12 16:55:45: Cropping tile = 75N060E
  |================================================================================================================================| 100%

2020-04-12 16:55:49: Masking the raster 
2020-04-12 16:55:49: Mask using rasterize 
2020-04-12 16:56:26: Writing the raster to disk 
  |================================================================================================================================| 100%

2020-04-12 16:56:29: Crop and mask using rasterize ... Done
2020-04-12 16:56:29: Begin extracting data from the raster 
  |================================================================================================================================| 100%
2020-04-12 16:56:39: DONE processing MMR 201512
2020-04-12 16:56:39: **COMPLETE. Writing data to disk
2020-04-12 16:56:39: **** COMPLETED PROCESSING :ctryCodes=MMR, admLevels=gadm36_MMR_1, nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201501", "201502", "201503", "201504", "201505", "201506", "201507", "201508", "201509", "201510", "201511", "201512"), nlStats=list(list("sum", na.rm = TRUE), list("median", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2020-04-12 16:56:40: All stats exist
2020-04-12 16:56:40: Retrieving requested data
Warning messages:
1: In RGEOSBinPredFunc(spgeom1, spgeom2, byid, func) :
  spgeom1 and spgeom2 have different proj4 strings
2: In RGEOSBinPredFunc(spgeom1, spgeom2, byid, func) :
  spgeom1 and spgeom2 have different proj4 strings
3: In RGEOSBinPredFunc(spgeom1, spgeom2, byid, func) :
  spgeom1 and spgeom2 have different proj4 strings
> #Optionally plot the data
> library(ggplot2)
> library(plotly)
> #melt the stats into key-value format for easy multi-line plotting with ggplot2
> highestAdmLevelStats <- melt(highestAdmLevelStats,
+                              id.vars = grep("NL_", names(highestAdmLevelStats), 
+                                             invert=TRUE), 
+                              variable.name = "nlPeriod", 
+                              value.name = "radiancesum")
> #extract date from the NL col names
> highestAdmLevelStats$nlPeriod <- substr(highestAdmLevelStats$nlPeriod, 12, 17)
> #format period as date
> highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4), 
+                                             "-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01"))
Warning message:
All formats failed to parse. No formats found. 
> #Optionally plot the data
> library(ggplot2)
> library(plotly)
> 
> #melt the stats into key-value format for easy multi-line plotting with ggplot2
> highestAdmLevelStats <- melt(highestAdmLevelStats,
+                              id.vars = grep("NL_", names(highestAdmLevelStats), 
+                                             invert=TRUE), 
+                              variable.name = "nlPeriod", 
+                              value.name = "radiancesum")
> #extract date from the NL col names
> highestAdmLevelStats$nlPeriod <- substr(highestAdmLevelStats$nlPeriod, 12, 17)
> 
> #format period as date
> highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4), 
+                                             "-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01"))
Warning message:
All formats failed to parse. No formats found. 
> #plot admin level sums for the year
> g <- ggplot(data = highestAdmLevelStats, 
+             aes(x=nlPeriod, y=radiancesum, 
+                 color=highestAdmLevelStats[[2]])) +
+   scale_x_date(date_breaks = "1 month", date_labels = "%Y-%m")+
+   geom_line()+geom_point() + labs(color = names(highestAdmLevelStats)[2]) + 
+   xlab("Month") + 
+   ylab("Sum of Radiances") +
+   ggtitle(paste0(unique(names(highestAdmLevelStats)[2]), " sum of radiances for ", ctry))
> 
> print(g)
Error in seq.int(r1$mon, 12 * (to0$year - r1$year) + to0$mon, by) : 
  'from' must be a finite number
In addition: Warning messages:
1: Use of `highestAdmLevelStats[[2]]` is discouraged. Use `.data[[2]]` instead. 
2: Use of `highestAdmLevelStats[[2]]` is discouraged. Use `.data[[2]]` instead. 
chrisvwn commented 4 years ago

Hi @HaasL

Seems something to do with the date formating. Could you post a sample of the the dates i.e. head(highestAdmLevelStats$nlPeriod)

firatgundem commented 4 years ago

I have the same errors:

> highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4), 
>        +  "-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01"))
Warning message:
All formats failed to parse. No formats found. 

And this:

g <- ggplot(data = highestAdmLevelStats, 
+             aes(x=nlPeriod, y=radiancesum, 
+                 color=highestAdmLevelStats[[2]])) +
+   scale_x_date(date_breaks = "1 month", date_labels = "%Y-%m")+
+   geom_line()+geom_point() + labs(color = names(highestAdmLevelStats)[2]) + 
+   xlab("Month") + 
+   ylab("Sum of Radiances") +
+   ggtitle(paste0(unique(names(highestAdmLevelStats)[2]), " sum of radiances for ", ctry))
 print(g)
Error in seq.int(r1$mon, 12 * (to0$year - r1$year) + to0$mon, by) : 
  'from' must be a finite number
In addition: Warning messages:
1: Use of `highestAdmLevelStats[[2]]` is discouraged. Use `.data[[2]]` instead. 
2: Use of `highestAdmLevelStats[[2]]` is discouraged. Use `.data[[2]]` instead

Also, I got this when I followed up with your suggestion:

head(highestAdmLevelStats$nlPeriod)
[1] NA

How can I fix this? Could you please help me?

chrisvwn commented 4 years ago

Hi @firatgundem

Which version of Rnightlights are you using? Also, could you post the whole set of code that you are running?

firatgundem commented 4 years ago

Since the Rnightlights package is not available anymore in CRAN, I used the code below: packageurl <- "https://cran.r-project.org/src/contrib/Archive/Rnightlights/Rnightlights_0.2.4.tar.gz"

install.packages(packageurl, repos=NULL, type="source") library(Rnightlights) library(dplyr) library(lubridate) library(reshape2)

pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)

ctry <- "KEN"

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", nlPeriods = nlRange("201401", "201412","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

library(ggplot2) install.packages("plotly") library(plotly)

highestAdmLevelStats <- melt(highestAdmLevelStats, id.vars = grep("NL_", names("highestAdmLevelStats"), invert=TRUE), variable.name = "nlPeriod", value.name = "radiancesum")

highestAdmLevelStats$nlPeriod <- substr(highestAdmLevelStats$nlPeriod, 12, 17)

highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4), "-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01")) Here I got this error: Warning message: All formats failed to parse. No formats found.

I kept running the rest of the code..

plot 2nd admin level sums for the year

g <- ggplot(data = highestAdmLevelStats, aes(x=nlPeriod, y=radiancesum, color=highestAdmLevelStats[[2]])) + scale_x_date(date_breaks = "1 month", date_labels = "%Y-%m")+ geom_line()+geom_point() + labs(color = names(highestAdmLevelStats)[2]) + xlab("Month") + ylab("Sum of Radiances") + ggtitle(paste0(unique(names(highestAdmLevelStats)[2]), " sum of radiances for ", ctry))

Then I got this error: Error in seq.int(r1$mon, 12 * (to0$year - r1$year) + to0$mon, by) : 'from' must be a finite number In addition: Warning messages: 1: Use of highestAdmLevelStats[[2]] is discouraged. Use .data[[2]] instead. 2: Use of highestAdmLevelStats[[2]] is discouraged. Use .data[[2]] instead.

I followed the step that you suggested to @HaasL and then I got this as well: [1] NA

Am I doing something wrong? I am learning R very new I would appreciate if you can help me. Thanks.

firatgundem commented 4 years ago

Also, now I am getting this error as I was trying to fix the problem I posted above:

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", nlPeriods = nlRange("201401", "201412","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

File'C:\Users\firat\AppData\Local\Temp\RtmpWS1kS1/.Rnightlights/data/NL_DATA_KEN_ADM1_GADM-3.6-SHPZIP.csv' does not exist or is non-readable. getwd()=='C:/Users/firat/Desktop/r_demo/lights' In addition: Warning message: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform

I am trying to fix these for a long time but could not proceed. Could you please help me?

firatgundem commented 4 years ago

I am trying to fix the errors that I got but I am seeing a different error in each attempt. Do you have any idea what's going on here? Could you please help me?

Error in proj4string<-(*tmp*, value = crs) : Geographical CRS given to non-conformant data: -1640.8982 2197.5138 In addition: Warning messages: 1: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform 2: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform

chrisvwn commented 4 years ago

Since the Rnightlights package is not available anymore in CRAN, I used the code below: packageurl <- "https://cran.r-project.org/src/contrib/Archive/Rnightlights/Rnightlights_0.2.4.tar.gz"

install.packages(packageurl, repos=NULL, type="source") library(Rnightlights) library(dplyr) library(lubridate) library(reshape2)

pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)

ctry <- "KEN"

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", nlPeriods = nlRange("201401", "201412","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

library(ggplot2) install.packages("plotly") library(plotly)

highestAdmLevelStats <- melt(highestAdmLevelStats, id.vars = grep("NL_", names("highestAdmLevelStats"), invert=TRUE), variable.name = "nlPeriod", value.name = "radiancesum")

highestAdmLevelStats$nlPeriod <- substr(highestAdmLevelStats$nlPeriod, 12, 17)

highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4), "-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01")) Here I got this error: Warning message: All formats failed to parse. No formats found.

I kept running the rest of the code..

plot 2nd admin level sums for the year

g <- ggplot(data = highestAdmLevelStats, aes(x=nlPeriod, y=radiancesum, color=highestAdmLevelStats[[2]])) + scale_x_date(date_breaks = "1 month", date_labels = "%Y-%m")+ geom_line()+geom_point() + labs(color = names(highestAdmLevelStats)[2]) + xlab("Month") + ylab("Sum of Radiances") + ggtitle(paste0(unique(names(highestAdmLevelStats)[2]), " sum of radiances for ", ctry))

Then I got this error: Error in seq.int(r1$mon, 12 * (to0$year - r1$year) + to0$mon, by) : 'from' must be a finite number In addition: Warning messages: 1: Use of highestAdmLevelStats[[2]] is discouraged. Use .data[[2]] instead. 2: Use of highestAdmLevelStats[[2]] is discouraged. Use .data[[2]] instead.

I followed the step that you suggested to @HaasL and then I got this as well: [1] NA

Am I doing something wrong? I am learning R very new I would appreciate if you can help me. Thanks.

If the formats failed to parse then the rest of the code will not work. This example is not the best code as there are hardcoded parts especially the substring which I believe is the cause of this problem.

chrisvwn commented 4 years ago

I am trying to fix the errors that I got but I am seeing a different error in each attempt. Do you have any idea what's going on here? Could you please help me?

Error in proj4string<-(*tmp*, value = crs) : Geographical CRS given to non-conformant data: -1640.8982 2197.5138 In addition: Warning messages: 1: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform 2: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform

I am not sure how you are trying to fix the errors, could you clarify? However, the main thing as pointed out above is to pull out the date here:

highestAdmLevelStats$nlPeriod <- substr(highestAdmLevelStats$nlPeriod, 12, 17)

highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4),
"-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01"))

This is where you need to look. Could you print out highestAdmLevelStats$nlPeriod before and after the substring? Then rewrite the substr command to pick the date correctly.

firatgundem commented 4 years ago

I am trying to fix the errors that I got but I am seeing a different error in each attempt. Do you have any idea what's going on here? Could you please help me? Error in proj4string<-(*tmp*, value = crs) : Geographical CRS given to non-conformant data: -1640.8982 2197.5138 In addition: Warning messages: 1: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform 2: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform

I am not sure how you are trying to fix the errors, could you clarify? However, the main thing as pointed out above is to pull out the date from the column names

I just deleted and re-uploaded things. As I sad you before, I am very new in R and I could not run the codes even for once. What should I do? Thanks!

chrisvwn commented 4 years ago

If you want to start afresh:

  1. Uninstall the package

    getCtryNlDataPath()
    remove.packages("Rnightlights")
  2. Delete the .Rnightlights folder where you created it at install time (The first command above should give you the path)

  3. Reinstall the package as before

chrisvwn commented 4 years ago

Closing this issue for now. If you would like to pursue this further please reopen.