search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Error while downloading nightlight data for 2021 #75

Open naiktejashvi opened 3 years ago

naiktejashvi commented 3 years ago

Hi Chris,

I am unable to download the data for year 2021, its not redirecting to the source https://eogdata.mines.edu/nighttime_light/monthly/v10/2021/. can you please suggest the solution?

chrisvwn commented 3 years ago

Hi @naiktejashvi ,

Apologies for the late response. Could you post the code you are running.

naiktejashvi commented 3 years ago

Hi Chris,

Thank you for your response. Please find the code as requested : library(Rnightlights) library(reshape2) library(lubridate) library(rgdal) library(httr)

Rnightlights:::nlCleanup()

file.remove(list.files(path = getNlDir("dirNlTemp"),pattern = "*.html",full.names = T))

ctry <- "IND" highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "admin3", nlType = "VIIRS.M", nlPeriods = nlRange("202101","202102","VIIRS.M"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

chrisvwn commented 3 years ago

Please paste a printout of the whole output.

chrisvwn commented 3 years ago

Actually, it seems the pages have changed and the code will need to be updated to reflect this. However, I may not be able to fix this any time soon as this will likely be time consuming.

naiktejashvi commented 3 years ago

Hi Chris,

Thank you for the immediate response. please find the output as below:

Processing missing data: IND:VIIRS.M:VCMCFG:avg_rade9:202101:sum, VIIRS.M:VCMCFG:avg_rade9:202102:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2021-08-10 15:52:59: START PROCESSING: ctryCodes=IND, admLevels=list(IND = "gadm36_IND_3"), nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("202101", "202102"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast 2021-08-10 15:52:59: Downloading country polygons ... | | 0%2021-08-10 15:52:59: Downloading polygon: IND 2021-08-10 15:52:59: Downloading ctry poly: IND 2021-08-10 15:52:59: Downloading ctry shpZip: IND 2021-08-10 15:52:59: Polygon dir for IND:3.6 already exists |==============================================================================| 100% 2021-08-10 15:52:59: Downloading country polygons ... DONE 2021-08-10 15:53:00: PROCESSING nlType:VIIRS.M | configName: VCMCFG | extension: avg_rade9 | nlPeriod:202101 2021-08-10 15:53:00: Checking tiles required for VIIRS.M 202101 2021-08-10 15:53:00: IND: Stats missing. Adding tiles 2021-08-10 15:53:00: numTiles: 1, Required tiles: 75N060E 2021-08-10 15:53:00: Downloading tile (1/1): 2021013 2021-08-10 15:53:03: Tile not available on the NOAA page. Please manually check for the 202101: Tile3_75N060E: VCMCFG at https://eogdata.mines.edu/download_dnb_composites.html'. If it exists please report this as a bug 2021-08-10 15:53:03: Something went wrong with the tile downloads. Aborting ... 2021-08-10 15:53:03: COMPLETED PROCESSING :ctryCodes=IND, admLevels=gadm36_IND_3, nlTypes=VIIRS.M, configNames=VCMCFG, extensions=avg_rade9, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlaresMethod=VTM, nlPeriods=c("202101", "202102"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast Error in data.table::fread(getCtryNlDataFnamePath(ctryCode = ctryCode, : File 'D:/Rnightlights/.Rnightlights/data/NL_DATA_IND_ADM3_GADM-3.6-SHPZIP.csv' does not exist or is non-readable.

chrisvwn commented 3 years ago

Confirmed. The nightlights index page structure has changed. The package scrapes the upstream tile list to get a list of available tiles. This will take a while to fix as there are likely other major changes that need to be addressed.

naiktejashvi commented 3 years ago

Hi Chris,

Thanks for the update. Please let me know, if in future you update the package and are able to extract the same.

chrisvwn commented 3 years ago

Will do