search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

problem with acessing VIIRS.Y #28

Closed neija11 closed 5 years ago

neija11 commented 5 years ago

Hi,

I am running into a problem with downloading the tiles from VIIRS.Y for 2015. Here is my code:

`getCtryNlData(ctryCode = "MDG", admLevel = "gadm36_MDG_0", nlType = "VIIRS.Y", nlPeriods = "2015", nlStats = list("mean",na.rm=TRUE), ignoreMissing=FALSE)

Processing missing data: MDG:VIIRS.Y:2015:mean. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2019-03-27 11:43:42: Downloading country polygons ... 2019-03-27 11:43:42: Downloading polygon: MDG 2019-03-27 11:43:42: Downloading ctry poly: MDG 2019-03-27 11:43:42: Polygon dir for MDG:3.6 already exists 2019-03-27 11:43:43: Downloading country polygons ... DONE 2019-03-27 11:43:43: PROCESSING nlType:VIIRS.Y nlPeriod:2015 2019-03-27 11:43:43: Checking tiles required for VIIRS.Y 2015 2019-03-27 11:43:43: MDG: Stats missing. Adding tiles 2019-03-27 11:43:441 Required tiles: 00N060W 2019-03-27 11:43:44: Downloading tile: 20155 2019-03-27 11:43:44: Tile not available on the NOAA page. Please manually check for the 2015 tile at 'https://ngdc.noaa.gov/eog/viirs/download_dnb_composites.html'. If it exists please report this as a bug 2019-03-27 11:43:44: Something went wrong with the tile downloads. Aborting ... Error in data.table::fread(getCtryNlDataFnamePath(ctryCode = ctryCode, : File 'C:/Users/Desktop/test//.Rnightlights/data/NL_DATA_MDG_ADM0_GADM-3.6.csv' does not exist or is non-readable. getwd()=='C:/Users/Desktop/test'`

It seems like it cannot access the tiles online, however checking the website, annual tiles exist for both 2015 and 2016. Same problem occurs when trying with other countries

Hope you can help!

chrisvwn commented 5 years ago

Hi @neija11 ,

Is it possible that your internet connection might have had a temporary outage? It works for me here. Please see the screenshot below:

Screenshot from 2019-03-27 14-35-00

From your message you can access the page directly so maybe the index page was incorrectly retrieved at that point. You could retry the command and it should overwrite it but you can try manually deleting the index page to force it to be re-retrieved by running:

file.remove(list.files(path = getNlDir("dirNlTemp"), pattern = "*.html", full.names = T))

HTH

neija11 commented 5 years ago

It seems like i am still having problems downloading the tiles. Internet is working fine, and i have no problems downloading the shapefiles for the administrative borders.

image

i also tried with the OLS data, but have similar problems with downloading.

chrisvwn commented 5 years ago

These are related to incomplete tile downloads. See the line that says An error occurred downloading I think we need to clear the cache. Could you also delete the files under .Rnightlights/tiles and also under .Rnightlights/outputrasters and try again?

neija11 commented 5 years ago

Yes, it seems like it was something to to with the already downloaded tiles. I started over with empty folders and now downloading works.

Thanks so much for the help!

chrisvwn commented 5 years ago

You're welcome and thanks for confirming.

chrisvwn commented 5 years ago

Closing issue

calvinke commented 4 years ago

I'm getting a similar issue. I tried to delete the files under /tiles and /outputrasters (as well as /polygons) but I get the same message:

2019-12-09 03:24:46: Downloading country polygons ... DONE 2019-12-09 03:24:47: **** PROCESSING nlType:VIIRS.Y | configName: VCM-ORM-NTL | nlPeriod:2015**** 2019-12-09 03:24:47: Checking tiles required for VIIRS.Y 2015 2019-12-09 03:24:47: IND: Stats missing. Adding tiles 2019-12-09 03:24:47: numTiles: 1, Required tiles: 75N060E 2019-12-09 03:24:47: Downloading tile: 20153 2019-12-09 03:24:47: An error occurred downloading 2019-12-09 03:24:47: Something went wrong with the tile downloads. Aborting ... 2019-12-09 03:24:47: **** COMPLETED PROCESSING :ctryCodes=IND, admLevels=0_IAGE71FL_Buffer, nlTypes=VIIRS.Y, configNames=VCM-ORM-NTL, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=2015, nlStats=sum, custPolyPath=C://Users/Documents/Data/alan/India.zip, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=aria, cropMaskMethod=gdal, extractMethod=gdal**** Error in getCtryNlData(ctryCode = "IND", admLevel = "adm0", custPolyPath = "C://Users/Documents/Data/alan/India.zip", : 2019-12-09 03:24:47: An error occurred

chrisvwn commented 4 years ago

Hi @calvinke I believe this is a different issue so I am starting a new thread for it