search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

VIIRS.Y Something went wrong with the tile downloads. Aborting ... #48

Closed chrisvwn closed 4 years ago

chrisvwn commented 4 years ago

I'm getting a similar issue. I tried to delete the files under /tiles and /outputrasters (as well as /polygons) but I get the same message:

2019-12-09 03:24:46: Downloading country polygons ... DONE 2019-12-09 03:24:47: **** PROCESSING nlType:VIIRS.Y | configName: VCM-ORM-NTL | nlPeriod:2015**** 2019-12-09 03:24:47: Checking tiles required for VIIRS.Y 2015 2019-12-09 03:24:47: IND: Stats missing. Adding tiles 2019-12-09 03:24:47: numTiles: 1, Required tiles: 75N060E 2019-12-09 03:24:47: Downloading tile: 20153 2019-12-09 03:24:47: An error occurred downloading 2019-12-09 03:24:47: Something went wrong with the tile downloads. Aborting ... 2019-12-09 03:24:47: **** COMPLETED PROCESSING :ctryCodes=IND, admLevels=0_IAGE71FL_Buffer, nlTypes=VIIRS.Y, configNames=VCM-ORM-NTL, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=2015, nlStats=sum, custPolyPath=C://Users/Documents/Data/alan/India.zip, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=aria, cropMaskMethod=gdal, extractMethod=gdal**** Error in getCtryNlData(ctryCode = "IND", admLevel = "adm0", custPolyPath = "C://Users/Documents/Data/alan/India.zip", : 2019-12-09 03:24:47: An error occurred

Originally posted by @calvinke in https://github.com/chrisvwn/Rnightlights/issues/28#issuecomment-563122105

chrisvwn commented 4 years ago

@calvinke Could you confirm that you have at least 4GB free on the drive on which .Rnightlights resides?

nreguera commented 4 years ago

Hi, it´s happening to me as well.

Untitled

I have enough space (have around 52gb)

calvinke commented 4 years ago

Hi Chris,

Yes I have over 55 GB free, I just ran it again (first clearing the contents of those folders mentioned above) and I still got the error:

` ctry <- "IND" getnldata_VIIRS<-getCtryNlData(ctryCode="IND", admLevel="adm0", custPolyPath="C://Users/Documents/Data/alan/India.zip", nlType="VIIRS.Y", nlPeriods="2015", nlStats="sum", ignoreMissing=FALSE)

2019-12-09 11:04:29: Downloading ctry poly: IND 2019-12-09 11:04:29: Downloading ctry shpZip: IND 2019-12-09 11:04:29: Downloading file:///C://Users/Documents/Data/alan/India.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 46.7M 100 46.7M 0 0 4528k 0 0:00:10 0:00:10 --:--:-- 2332k OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "IAGE71FL_Buffer" with 28526 features It has 24 fields Integer64 fields read as strings: ORIG_FID 2019-12-09 11:08:47: Creating integer zone attribute col for polygon 2019-12-09 11:09:01: Writing layer with new idx col OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "IAGE71FL_Buffer" with 28526 features It has 25 fields 2019-12-09 11:10:40: Saving shapefile as RDS for faster access 2019-12-09 11:10:40: Getting admLevels in IND 2019-12-09 11:10:40: Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\Users\Documents.Rnightlights\polygons\POLY_IND_CUST-India.zip-SHPZIP", layer: "0_IAGE71FL_Buffer" with 28526 features It has 25 fields 2019-12-09 11:11:58: Saving admLevel polygons as RDS Processing missing data: IND:VIIRS.Y:VCM-ORM-NTL:2015:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2019-12-09 11:12:16: START PROCESSING: ctryCodes=IND, admLevels=0_IAGE71FL_Buffer, nlTypes=VIIRS.Y, configNames=VCM-ORM-NTL, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=2015, nlStats=sum, custPolyPath=C://Users/Documents/Data/alan/India.zip, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=aria, cropMaskMethod=gdal, extractMethod=gdal 2019-12-09 11:12:16: Downloading country polygons ... | | 0%2019-12-09 11:12:17: Downloading polygon: IND 2019-12-09 11:12:17: Downloading ctry poly: IND 2019-12-09 11:12:17: Downloading ctry shpZip: IND 2019-12-09 11:12:17: Polygon dir for IND:India.zip already exists |============================================================================================================| 100% 2019-12-09 11:12:17: Downloading country polygons ... DONE 2019-12-09 11:12:17: PROCESSING nlType:VIIRS.Y | configName: VCM-ORM-NTL | nlPeriod:2015 2019-12-09 11:12:17: Checking tiles required for VIIRS.Y 2015 2019-12-09 11:12:17: IND: Stats missing. Adding tiles 2019-12-09 11:12:21: numTiles: 1, Required tiles: 75N060E 2019-12-09 11:12:21: Downloading tile: 20153 2019-12-09 11:12:24: An error occurred downloading 2019-12-09 11:12:24: Something went wrong with the tile downloads. Aborting ... 2019-12-09 11:12:24: COMPLETED PROCESSING :ctryCodes=IND, admLevels=0_IAGE71FL_Buffer, nlTypes=VIIRS.Y, configNames=VCM-ORM-NTL, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=2015, nlStats=sum, custPolyPath=C://Users/Documents/Data/alan/India.zip, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=aria, cropMaskMethod=gdal, extractMethod=gdal Error in getCtryNlData(ctryCode = "IND", admLevel = "adm0", custPolyPath = "C://Users/Documents/Data/alan/India.zip", : 2019-12-09 11:12:24: An error occurred `

chrisvwn commented 4 years ago

This is interesting considering your commands are somewhat different.

@nreguera if I am not wrong it worked for VIIRS.M 201801 and 201802? Then failed at 201803? And you are using downloadMethod='aria'

@calvinke if I am not wrong you are trying VIIRS.Y but it has not worked for any year yet?

chrisvwn commented 4 years ago

So I seem to have them working on my computer. I am thinking either this is a temporary problem with the upstream repository or something related to a previously failed download.

In both examples I have cancelled the download but you can see that they were progressing.

@nreguera 

> getCtryNlData(ctryCode = "VNM", admLevel = "admin3", nlTypes = "VIIRS.M",nlPeriods = nlRange("201803","201812"), ignoreMissing = F, nlStats=list("sum","na.rm=TRUE"))
2019-12-09 19:44:04: NlRange autodetected nlType: VIIRS.M
Processing missing data: VNM:VIIRS.M:VCMCFG:201803:sum, VIIRS.M:VCMCFG:201804:sum, VIIRS.M:VCMCFG:201805:sum, VIIRS.M:VCMCFG:201806:sum, VIIRS.M:VCMCFG:201807:sum, VIIRS.M:VCMCFG:201808:sum, VIIRS.M:VCMCFG:201809:sum, VIIRS.M:VCMCFG:201810:sum, VIIRS.M:VCMCFG:201811:sum, VIIRS.M:VCMCFG:201812:sum. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2019-12-09 19:44:05: **** START PROCESSING: ctryCodes=VNM, admLevels=list(VNM = "gadm36_VNM_3"), nlTypes=VIIRS.M, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("201803", "201804", "201805", "201806", "201807", "201808", "201809", "201810", "201811", "201812"), nlStats=list(list("sum", "na.rm=TRUE")), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=aria, cropMaskMethod=rast, extractMethod=rast****
2019-12-09 19:44:05: Downloading country polygons ...
2019-12-09 19:44:05: Downloading polygon: VNM
2019-12-09 19:44:05: Downloading ctry poly: VNM
2019-12-09 19:44:05: Downloading ctry shpZip: VNM
2019-12-09 19:44:05: Polygon dir for VNM:3.6 already exists
2019-12-09 19:44:05: Downloading country polygons ... DONE
2019-12-09 19:44:05: **** PROCESSING nlType:VIIRS.MconfigName: VCMCFG nlPeriod:201803****
2019-12-09 19:44:05: Checking tiles required for VIIRS.M 201803
2019-12-09 19:44:05: VNM: Stats missing. Adding tiles
2019-12-09 19:44:05: numTiles: 1, Required tiles: 75N060E
2019-12-09 19:44:05: Downloading tile: 2018033

12/09 19:44:05 [NOTICE] Downloading 1 item(s)

12/09 19:44:07 [NOTICE] Allocating disk space. Use --file-allocation=none to disable it. See --file-allocation option in man page for more details.
 *** Download Progress Summary as of Mon Dec  9 19:44:18 2019 *** 
===============================================================================
[#e4ea35 0B/470MiB(0%) CN:1 DL:0B]
FILE: /media/NewVolume/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201803_75N060E.tgz
-------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Dec  9 19:44:28 2019 *** 
===============================================================================
[#e4ea35 5.6MiB/470MiB(1%) CN:2 DL:582KiB ETA:13m37s]
FILE: /media/NewVolume/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201803_75N060E.tgz
-------------------------------------------------------------------------------

12/09 19:44:30 [NOTICE] Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.

12/09 19:44:30 [NOTICE] Download GID#e4ea3517016c22ef not complete: /media/NewVolume/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201803_75N060E.tgz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
e4ea35|INPR|   387KiB/s|/media/NewVolume/.Rnightlights/tiles/NL_TILE_VIIRS.M_VCMCFG_201803_75N060E.tgz

Status Legend:
(INPR):download in-progress.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

@calvinke 

> getCtryNlData(ctryCode = "IND", admLevel = "adm0", nlTypes = "VIIRS.Y",nlPeriods = "2015", ignoreMissing = F, nlStats=list("sum","na.rm=TRUE"), custPolyPath = "/media/NewVolume/Downloads/India.zip")
Processing missing data: IND:VIIRS.Y:VCM-ORM-NTL:2015:sum. This may take a while. 
Note: Set 'ignoreMissing=TRUE' to return only data found or 
'ignoreMissing=NULL' to return NULL if not all the data is found
2019-12-09 19:54:41: **** START PROCESSING: ctryCodes=IND, admLevels=0_IAGE71FL_Buffer, nlTypes=VIIRS.Y, configNames=VCM-ORM-NTL, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=2015, nlStats=list("sum", "na.rm=TRUE"), custPolyPath=/media/NewVolume/Downloads/India.zip, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=rast, extractMethod=rast****
2019-12-09 19:54:41: Downloading country polygons ...
2019-12-09 19:54:41: Downloading polygon: IND
2019-12-09 19:54:41: Downloading ctry poly: IND
2019-12-09 19:54:41: Downloading ctry shpZip: IND
2019-12-09 19:54:41: Polygon dir for IND:India.zip already exists
2019-12-09 19:54:41: Downloading country polygons ... DONE
2019-12-09 19:54:41: **** PROCESSING nlType:VIIRS.YconfigName: VCM-ORM-NTL nlPeriod:2015****
2019-12-09 19:54:41: Checking tiles required for VIIRS.Y 2015
2019-12-09 19:54:41: IND: Stats missing. Adding tiles
2019-12-09 19:54:41: numTiles: 1, Required tiles: 75N060E
2019-12-09 19:54:41: Downloading tile: 20153
trying URL 'https://eogdata.mines.edu/wwwdata/viirs_products/dnb_composites/v10//2015/SVDNB_npp_20150101-20151231_75N060E_v10_c201701311200.tgz'
Content type 'application/x-gzip' length 4147680700 bytes (3955.5 MB)
==

Could you try them again and let me know if there is any difference now?

nreguera commented 4 years ago

I tried again and its still not working. The only thing I can say is that this happens after reinstalling it. I set up 201801 to check that the package detected the last downloaded period (that was 201802) and started with the next one, 201803, as it did.

The "tiles" folder is empty btw.

chrisvwn commented 4 years ago

Thanks @nreguera I am quite stumped right now.

Would either you or @calvinke have teamviewer so I can connect and have a quick look around? This will likely be happening to others as well.

chrisvwn commented 4 years ago

Hmm. I have just noticed that your commands do not seem to be downloading the tile index pages. Could you try running Rnightlights:::nlCleanup() to empty the index from the temp cache and retry your function runs.

calvinke commented 4 years ago

Hi Chris, I'm getting the same error even after running nlCleanup. I have just downloaded and installed TeamViewer.

chrisvwn commented 4 years ago

Hi @calvinke

Could you send me Teamviewer details on email? chris.njuguna@gmail.com

calvinke commented 4 years ago

Done

chrisvwn commented 4 years ago

Thanks @calvinke Glad we sorted out that stage of the process. Issue was pkgOption(downloadMethod='aria') was set but the package couldn't find aria in the path. For now, we have set downloadMethod='auto' to use the internal downloader.

@nreguera I seem to remember you already have aria2c installed (right?). So, we may need a little more troubleshooting for your situation.

nreguera commented 4 years ago

The same happened to me somehow, I "reinstall" aria again and it looks it´s working...

chrisvwn commented 4 years ago

@nreguera Great! This is indeed an interesting case. Maybe it is related to something that happened in either aria or Windows.

Closing for now. Please reopen if the issue is not resolved or recurs.