Closed pk1196 closed 4 years ago
Hi,
Could you check that the polygon folder exists. You can find the path with:
library(Rnightlights)
getNlDir('dirPolygon')
If it does not exist please run: Rnightlights:::createNlDataDirs()
gadmPolyType='spRds'
hi Chris, thanks so much for your speedy response and for the heads up on the issue with the map. I checked on the polygon folder, and it does exist.
Hmm. Okay. What version of Rnightlights are you running?
0.2.4. this is the command I used to install it.
packageurl <- "https://cran.r-project.org/src/contrib/Archive/Rnightlights/Rnightlights_0.2.4.tar.gz" install.packages(packageurl, repos=NULL, type="source")
Could you try deleting any folder or file with IDN
in the name from the polygon folder we checked earlier, then try again?
I tried this earlier and it unfortunately did not work.
Okay. What is your polygon path as printed by the previous command?
No problem. I just needed to see the structure.
Seems like this might be a bit of a problem with unzip referencing network drives. Is xxx an IP address or hostname? I want to simulate a similar environment here.
No I think that's the name of the network drive
Okay. What is your current working directory as given by getwd()
?
C:/WINDOWS/system32
Okay. Thanks. This is the first time I am encountering this issue, so please bear with me.
Did you set the dataPath when you installed the package? Or did you run setupDataPath()
?
no worries, I am rather new to R so I'm very grateful for your help! no, I don't think I did either of those things. definitely not the latter.
Okay. So then the package was probably installed by someone else maybe the IT dept? I am asking to try to get a feel for how the package is accessing the data folder.
Also, could you try to run listNlTiles()
and listCtryNlData()
One more thing you can try is to map the drive to a local drive or folder and then set up this as the data path by running setupDataPath()
I installed the package but R and Rstudio were installed by the IT dept - not sure if the path could have been set then. I don't recall having to set any path while installing the package. I tried running listNlTiles() and listCtryNlData() and both return NULL. sorry if this is a silly question but how do I run setupDataPath() ? I am prompted to select 1 or 2 and I select 2 to choose a different directory but nothing happens after I type 2 and hit enter.
Ah my bad, the dialog box was frozen behind the Rstudio window. changed the location and that seems to have done the trick! thank you so much I really appreciate your time and help.
Great! I am glad that worked. Yes, sometimes the dialog box does that and this file/path selection bit needs some work. At some point I hope to look at whether network drives can be used directly.
Hi Chris, sorry to bother you again but I was in the process of downloading the tiles and got this error: 2019-12-05 17:18:55: PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201908 2019-12-05 17:18:55: Checking tiles required for VIIRS.M 201908 2019-12-05 17:18:55: IND: Stats missing. Adding tiles 2019-12-05 17:18:55: numTiles: 1, Required tiles: 75N060E 2019-12-05 17:18:55: Downloading tile: 2019083 2019-12-05 17:18:55: Tile not available on the NOAA page. Please manually check for the 201908 tile at 'https://ngdc.noaa.gov/eog/viirs/download_dnb_composites.html'. If it exists please report this as a bug 2019-12-05 17:18:55: Something went wrong with the tile downloads. Aborting ... Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2019-12-05 17:18:57: An error occurred
I went to the website provided to check for the tile and noticed that the most recent tile available was 201904, but I had managed to download up to 201907 before this error occurred. This website has been updated for the most recent tiles until 201909 https://eogdata.mines.edu/download_dnb_composites.html. Is it possible to extract them from here?
Another issue I am encountering is the formatting of the period as date. I get this error:
Warning message: All formats failed to parse. No formats found.
I just keyed in highestAdmLevelStats$nlperiod to check if this was what was causing the issue, and I got a bunch of NA values in return. The ymd function works fine otherwise when I replace highestAdmLevelStats$nlperiod with 201904 for example. Would really appreciate your help with this, thank you!
Is it possible to extract them from here?
The url changed after v0.2.4 went out so is only updated in the github version. Yes, you can update the link by running:
pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/download_dnb_composites_iframe.html")
Another issue I am encountering is the formatting of the period as date
Where are you receiving this error from? The nlPeriods do not qualify as dates so will not parse directly. Some manipulation is required. Essentially, for the monthly nlPeriods you can add a day date e.g. 01 at the end of the date and use ymd
as you are doing.
Looking at the example given on github this line does the formatting and conversion to date:
highestAdmLevelStats$nlPeriod <-
ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4),
"-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01"))
Running lubridate::ymd directly on an nlPeriod may produce inaccurate dates e.g. lubridate::ymd("201204")
gives "2020-12-04" instead of "2012-04-01".
Yes, however this is the line I am receiving that error from.
highestAdmLevelStats$nlPeriod <- ymd(paste0(substr(highestAdmLevelStats$nlPeriod, 1,4), "-",substr(highestAdmLevelStats$nlPeriod, 5,6), "-01"))
Oh I know why! So sorry. Since the names of the columns in the result have changed, the example is outdated. I need to update the documentation.
The reason why you're getting NAs is because this line is not pulling out the nlPeriod from the col name:
#extract date from the NL col names
highestAdmLevelStats$nlPeriod <- substr(highestAdmLevelStats$nlPeriod, 12, 17)
Try this instead:
#extract date from the NL col names
highestAdmLevelStats$nlPeriod <- stringr::str_extract(highestAdmLevelStats$nlPeriod, "\\d+")
Thank you very much, that solved the date issue. However, pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/download_dnb_composites_iframe.html") doesn't seem to work for the remaining downloads. I still get the same error
library(Rnightlights) library(lubridate) library(reshape2)
pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/download_dnb_composites_iframe.html")
ctry <- "IND" #replace to run for any other country
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", nlPeriods = nlRange("201204", "201909","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)
2019-12-06 11:49:58: PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201908 2019-12-06 11:49:58: Checking tiles required for VIIRS.M 201908 2019-12-06 11:49:58: IND: Stats missing. Adding tiles 2019-12-06 11:49:58: numTiles: 1, Required tiles: 75N060E 2019-12-06 11:49:58: Downloading tile: 2019083 2019-12-06 11:49:58: Tile not available on the NOAA page. Please manually check for the 201908 tile at 'https://ngdc.noaa.gov/eog/viirs/download_dnb_composites.html'. If it exists please report this as a bug 2019-12-06 11:49:58: Something went wrong with the tile downloads. Aborting ... Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2019-12-06 11:50:00: An error occurred
Also, just wanted to confirm - the csv file named NL_DATA_IND_ADM1_GADM-3.6-SHPZIP in the data folder contains the radiance sums for each state? Would it be right to say that adding all of them would give the radiance sum for the entire country in that month?
pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/download_dnb_composites_iframe.html") doesn't seem to work for the remaining downloads. I still get the same error
This is a different issue. The tile for 201908 has not yet been made available on the site. The error message needs to be updated with the new url to check.
Also, just wanted to confirm - the csv file named NL_DATA_IND_ADM1_GADM-3.6-SHPZIP in the data folder contains the radiance sums for each state? Would it be right to say that adding all of them would give the radiance sum for the entire country in that month?
Yes, the file contains the radiance sums for each member of the given admLevel.
Theoretically, it is right to assume that the sum of all the radiance sums at a lower level should equal the radiance sum at the next higher level. However, practically there may be a (very) small difference in sums. This may be attributed to slight variations in the polygon boundaries of the different admin levels.
Oh I think that's a different website, 201908 and 201909 are available here https://eogdata.mines.edu/download_dnb_composites.html
I see. You are right. It seems the page structure has changed.
Try setting it to this url:
pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")
That does not seem to work unfortunately, I still get the same error message.
Could you print your output? I seem to get it here
> pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html") > ctry <- "IND" #replace to run for any other country > > #download and process monthly VIIRS stats at the highest admin level > highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, + admLevel = "highest", + nlType = "VIIRS.M", + nlPeriods = "201908", + nlStats = list("sum",na.rm=TRUE), + ignoreMissing=FALSE) Processing missing data: IND:VIIRS.M:VCMCFG:201908:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2019-12-06 10:56:44: **** START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=201908, nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast**** 2019-12-06 10:56:44: Downloading country polygons ... 2019-12-06 10:56:44: Downloading polygon: IND 2019-12-06 10:56:44: Downloading ctry poly: IND 2019-12-06 10:56:44: Downloading ctry shpZip: IND 2019-12-06 10:56:44: Polygon dir for IND:3.6 already exists 2019-12-06 10:56:44: Downloading country polygons ... DONE 2019-12-06 10:56:44: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201908**** 2019-12-06 10:56:44: Checking tiles required for VIIRS.M 201908 2019-12-06 10:56:44: IND: Stats missing. Adding tiles 2019-12-06 10:56:44: numTiles: 1, Required tiles: 75N060E 2019-12-06 10:56:44: Downloading tile: 2019083 trying URL 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html' Content type 'text/html; charset=UTF-8' length 239739 bytes (234 KB) ================================================== downloaded 234 KB trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//201908/vcmcfg/SVDNB_npp_20190801-20190831_75N060E_vcmcfg_v10_c201909051300.tgz' Content type 'application/x-gzip' length 371652391 bytes (354.4 MB)
Sure, this is what I get:
pkgOptions(ntLtsIndexUrlVIIRS.M="https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")
(Optional performance enhancement if you have aria2c and gdal installed)
pkgOptions(downloadMethod = "aria", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)
Optional performance enhancement. If extractMethod="rast" you can specify the number of
CPU cores to use in parallel
pkgOptions(extractMethod = "rast", numCores=4)
ctry <- "IND" #replace to run for any other country
download and process monthly VIIRS stats at the highest admin level
highestAdmLevelStats <- getCtryNlData(ctryCode = ctry,
- admLevel = "highest",
- nlType = "VIIRS.M",
- nlPeriods = nlRange("201908", "201909","VIIRS.M"),
- nlStats = list("sum",na.rm=TRUE),
- ignoreMissing=FALSE) Processing missing data: IND:VIIRS.M:VCMCFG:201908:sum, VIIRS.M:VCMCFG:201909:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2019-12-06 15:52:38: START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_1"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201908", "201909"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2019-12-06 15:52:38: Downloading country polygons ... 2019-12-06 15:52:38: Downloading polygon: IND 2019-12-06 15:52:38: Downloading ctry poly: IND 2019-12-06 15:52:38: Downloading ctry shpZip: IND 2019-12-06 15:52:38: Polygon dir for IND:3.6 already exists 2019-12-06 15:52:38: Downloading country polygons ... DONE 2019-12-06 15:52:38: PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201908 2019-12-06 15:52:38: Checking tiles required for VIIRS.M 201908 2019-12-06 15:52:38: IND: Stats missing. Adding tiles 2019-12-06 15:52:38: numTiles: 1, Required tiles: 75N060E 2019-12-06 15:52:38: Downloading tile: 2019083 2019-12-06 15:52:38: Tile not available on the NOAA page. Please manually check for the 201908 tile at 'https://ngdc.noaa.gov/eog/viirs/download_dnb_composites.html'. If it exists please report this as a bug 2019-12-06 15:52:38: Something went wrong with the tile downloads. Aborting ... Error in getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.M", : 2019-12-06 15:52:38: An error occurred
Ah okay. The problem is that the index page is cached. Please run Rnightlights:::nlCleanup()
to clear the temp cache.
Ah that solves it, thanks very much for your help!
You're welcome!
hi, i get the following error when i try to run this for IDN:
Error in utils::unzip(zipfile = polyFnameZip, junkpaths = TRUE, exdir = polyFnamePath) : 'exdir' does not exist