An error in downloading VIIRS data

[fnarita]

Hi Chris,

I have tried the following code:

ctry <- "UGA" highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "country", nlType = "VIIRS.Y", nlPeriods = nlRange("2015", "2015"), nlStats = "sum", ignoreMissing=FALSE)

and received the following log with an error at the end (in the bold font):

Processing missing data: UGA:VIIRS.Y:2015:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2018-08-23 16:15:02: Downloading country polygons ... 2018-08-23 16:15:02: Downloading polygon: UGA 2018-08-23 16:15:02: Downloading ctry poly: UGA 2018-08-23 16:15:02: Polygon dir for UGA already exists 2018-08-23 16:15:02: Downloading country polygons ... DONE 2018-08-23 16:15:03: PROCESSING nlType:VIIRS.Y nlPeriod:2015 2018-08-23 16:15:03: Checking tiles required for VIIRS.Y 2015 2018-08-23 16:15:03UGA: Stats missing. Adding tiles 2018-08-23 16:15:042 Required tiles: 75N060W,00N060W 2018-08-23 16:15:04: Downloading tile: 20152 trying URL 'https://data.ngdc.noaa.gov/instruments/remote-sensing/passive/spectrometers-radiometers/imaging/viirs/dnb_composites/v10//2015/SVDNB_npp_20150101-20151231_75N060W_v10_c201701311200.tgz' Content type 'application/x-gzip' length 4156739880 bytes (3964.2 MB) downloaded 3964.2 MB

2018-08-23 16:43:33: Extracting C:/R/data/.Rnightlights/tiles/NL_TILE_VIIRS.Y_2015_75N060W.tgz 2018-08-23 16:43:33: Getting list of files in C:/R/data/.Rnightlights/tiles/NL_TILE_VIIRS.Y_2015_75N060W.tgz 2018-08-23 16:45:53: Decompressing SVDNB_npp_20150101-20151231_75N060W_vcm-ntl_v10_c201701311200.avg_rade9.tifSVDNB_npp_20150101-20151231_75N060W_vcm-orm-ntl_v10_c201701311200.avg_rade9.tifSVDNB_npp_20150101-20151231_75N060W_vcm-orm_v10_c201701311200.avg_rade9.tifSVDNB_npp_20150101-20151231_75N060W_vcm_v10_c201701311200.avg_rade9.tif Error in file.rename(file.path(getNlDir("dirNlTiles"), tgzAvgRadFilename), : 'from' and 'to' are of different lengths

I also noticed that the code downloads "NL_TILE_VIIRS.Y_2015_75N060W.tgz"--3 GB file--again, even after it downloaded it once, although this is not the case for the OLS data.

Could you take a look on these?

Thank you very much, Futoshi

[chrisvwn]

Hi Futoshi,

Let me have a look and get back to you shortly. Looks like the contents of the VIIRS.Y tar.gz might have changed.

Regards,

Chris

[chrisvwn]

This is now fixed. The problem was the regex identifying which file to extract from the compressed .tgz was matching more than one file when it could only handle one. Since this error was happening after the .tar.gz has been decompressed (thus destroying the .tar.gz) but before the extraction of tifs, at the next run the package started the download afresh.

Please update from github using:

devtools::install_github("chrisvwn/Rnightlights")

It should be on CRAN soon.

[fnarita]

Hi Chris,

Sorry for my belated response. Now I have confirmed that this is fixed and that I am able to download the 2015 VIIRS nighttime lights data!

I agree that the VIIRS data files are differently deployed and now see that the annual 2016 data are also available. It would be interesting to see how annual data compare with averaged monthly data, which is my motivation for this exercise.

Thank you very much for your very quick fix! Please feel free to close this issue.

Thank you! Futoshi

[fnarita]

Sorry Chris, but now I got a related error when I tried to download monthly data by the following code:

ctry <- "UGA" highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "country", nlType = "VIIRS.M", nlPeriods = nlRange("201501", "201512"), nlStats = "sum", ignoreMissing=FALSE)

The code successfully downloads (and does not repeat once it downloads) the image file, but after that, it gave me a log as below, which an error at the end:

Processing missing data: UGA:VIIRS.M:201501:sum, VIIRS.M:201502:sum, VIIRS.M:201503:sum, VIIRS.M:201504:sum, VIIRS.M:201505:sum, VIIRS.M:201506:sum, VIIRS.M:201507:sum, VIIRS.M:201508:sum, VIIRS.M:201509:sum, VIIRS.M:201510:sum, VIIRS.M:201511:sum, VIIRS.M:201512:sum. This may take a while. Note: Set 'ignoreMissing=TRUE' to return only data found or 'ignoreMissing=NULL' to return NULL if not all the data is found 2018-08-30 15:38:39: Downloading country polygons ... 2018-08-30 15:38:39: Downloading polygon: UGA 2018-08-30 15:38:39: Downloading ctry poly: UGA 2018-08-30 15:38:39: Polygon dir for UGA already exists 2018-08-30 15:38:39: Downloading country polygons ... DONE 2018-08-30 15:38:40: PROCESSING nlType:VIIRS.M nlPeriod:201501 2018-08-30 15:38:40: Checking tiles required for VIIRS.M 201501 2018-08-30 15:38:41UGA: Stats missing. Adding tiles 2018-08-30 15:38:412 Required tiles: 75N060W,00N060W 2018-08-30 15:38:41: Cropped raster C:/R/data/.Rnightlights/outputrasters/NL_UGA_VIIRS.M_201501_GADM-2.8.tif already exists. Skipping tile download 2018-08-30 15:38:41: processNLCountry: UGA UGA_adm0 VIIRS.M 201501 2018-08-30 15:38:41: Check for existing data file 2018-08-30 15:38:41: Data file found: C:/R/data/.Rnightlights/data/NL_DATA_UGA_ADM0_GADM-2.8.csv 2018-08-30 15:38:42: Processing stats: sum 2018-08-30 15:38:42: Load country data file 2018-08-30 15:38:42: Load country polygon admin level 2018-08-30 15:38:43: Begin extracting the data from the merged raster |=====================================================================================| 100% 2018-08-30 15:41:04: DONE processing UGA 201501 2018-08-30 15:41:04: COMPLETE. Writing data to disk Error in file(file, ifelse(append, "a", "w")) : cannot open the connection

I thought I should have opened a separate issue, but this error could be related, so I decided to ask you at this thread.

I would very much appreciate it if you could take a look on this too.

Futoshi

[fnarita]

Oops, Chris, this is just because I was opening the result csv file, preventing the code to modify the file!

So, there is no issue now. Sorry for the mess!

I have now realized that I can close the issue by myself, so I will close this right away.

Thank you very much Chris!

Futoshi

[chrisvwn]

Hi Futoshi,

You're welcome and thank you for catching the change with the annual raster download.

While preparing to deploy these changes to CRAN one of the package tests failed related to the function calculation output. There is a change I am making with the function calculation capabilities and the github code might be unstable right now. I thought I should let you know in case you get differences in your output. I am looking into this.

Thanks,

Chris

[fnarita]

Thank you very much Chris for letting me know! This is very helpful and it is very nice of you to notify that to me.

Hope the issue you face will be resolved soon.

Have a nice weekend, Futoshi

[chrisvwn]

Just an update the function should work fine. The test was outdated and needed to be updated to the latest format.

Thanks

[fnarita]

Thank you very much Chris for the nice update! This is good news!

Thank you! Futoshi