Toward closer replication of previous work

[fnarita]

Dear Chris,

This is such a great package and I'm so grateful to you for creating it! Thank you so much!

I have two feature requests and one question as follows.

1. On DMSP-OLS data, there are often more than one satellite providing the image for one year (e.g., F101994 and F121994). Would it be possible to have an option to process both images if there are more than one? This is motivated by the fact that a published paper ("Measuring Economic Growth from Outer Space" by Henderson,Storeygard, and Weil, 2012) uses the average across satellites. In this package, it seems to first identify the file to download by searching for a format like F*yyyy and I guess the code picks up the first one (e.g., F101994 for 1994).

2. Would it be possible to process the data only at the country level? I set admLevel = "Country" but it seems to run at the highest adm. level. I can sum up afterward, but it may be useful for those who only look at the country-level data.

3. I sometimes receive an error for small countries such as Kiribati. Do you have any guess on the error? My guess is that it would be because these countries are too small. For your information, my environment is Win 10, R3.4.3, and your package version is 0.2.2. The code that I ran was a loop version of the following:

   
   library(Rnightlights)
   library(lubridate)
   library(reshape2)

setupDataPath("C:/R/data")

ctry <- "KIR" highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "Country", nlType = "OLS.Y", nlPeriods = nlRange("2008", "2013"), nlStats = "sum", ignoreMissing=FALSE)

... and below is the log:

Downloading ctry poly: KIR trying URL 'http://biogeo.ucdavis.edu/data/gadm2.8/shp/KIR_adm_shp.zip' Content type 'application/zip' length 205243 bytes (200 KB) downloaded 200 KB

Saving shapefile as RDS for faster access Getting admLevels in KIR Reading in all admLevels OGR data source with driver: ESRI Shapefile Source: "C:\R\data.Rnightlights\polygons\SHP_KIR_GADM-2.8", layer: "KIR_adm0" with 1 features It has 67 fields Integer64 fields read as strings: ID_0 Saving admLevel polygons as RDS Saving country admLevel structure to CSV Error in [.data.frame(ctryPoly@data, , eval(ctryPolyAdmCols)) : undefined columns selected

By the way, I'm very new to both GitHub and R, as I just started on Tuesday (July 31). It is amazing that, even so, I have successfully downloaded nightlight data already for many countries, purely thanks to your package.

Thank you!
Futoshi

[chrisvwn]

Hi Futoshi,

Welcome to R and Github! You see to be doing well in them already!

In response to your message:

1. You are right. I have been meaning to correct this behaviour. Yes, with OLS calculations should use a mean of the tiles where there are more than one. Yes, right now it just uses the first tile. I will correct this shortly.

2. I have actually just fixed a bug that was caused by a case-sensitive match for the word "country". The update is only on github for now so you need to install this version using this code:

install.packages("devtools")
devtools::install_github("chrisvwn/Rnightlights")

3. Regarding small countries, I suspect that any errors you are experiencing are related to the bug in point 2 above. Please try again after updating to the dev version.

All the best!

[fnarita]

Hi Chris,

Thank you so much for your very fast and helpful response! My R and GitHub life has started very well, totally thanks to you!

On 1., it sounds great! Thank you for your ongoing efforts!

On 2 and 3., yes, now it works! Thank you very much for the patch!

One follow-up on this is that now the resulting csv file shows a country's iso 3 code (e.g., KIR or UGA), instead of a country name (e.g., Kiribati or Uganda). It would be fine if this is an intended change, but would like to flag it to you just in case.

Another point is that now I have realized that area_sq_km indicates a slightly different size of countries from Wikipedia or the data file by Henderson, Storeygard, and Weil (HSW, 2012, variable "areasqkm").

	Kiribati	Uganda
Rnightlights:	1,012.082	241,497.9
Wikipedia:	811.000	241,038.0
HSW (2012):	706.320	206,986.2

I don't know much about Wikipedia nor the data from HSW (2012), while I plan to read the data compile section of their paper more carefully later, but do you have any guess? It could be due to any differences in adjusting "map area" into "real area" by taking into account the map projection with any assumption on the shape of the globe. HSW (2012) uses ArcGIS Desktop and ArcInfo Workstation GIS software, just for your information.

By the way, I don't mean to precisely replicate the above-mentioned paper's results, but rather would like to list any possible reasons for any differences.

Thank you very much again, and have a nice weekend!

Futoshi

[chrisvwn]

Thank you for the feedback! I am glad 1 and 2 got sorted out.

On the ISO3 code in the file, that is actually the intention. I think it is a bit more precise since countries can have multiple names.

On the area, this is only for convenience and not to be taken as an authority. It is calculated from the given polygons. Multiple issues could come up such as: i) the polygons are not official (see: GADM) and ii) the date of release of the polygons may vary from the dates when areas were calculated by the other sources. Also as you rightly point out the projection in use matters.

I must hasten to add that I am not a GIS expert so this package is provided purely as a convenience.

[fnarita]

Thank you very much Chris for your quick and very helpful response! I very much appreciate your explanations. I’m not a GIS expert either but you are already great! Hope I could learn more and possibly contribute.

Have a nice weekend, Futoshi

[chrisvwn]

Thank you Futoshi. Contributions are most welcome! Have a great weekend.