Decipher data files

[nreguera]

Hello Chris,

I have been downloading data for a while, and now I am realizing that I don´t understand very well all the types of files that I have accumulated using Rnightlights. This is how currently looks my local data folder:

Can you help me identify which files do I need and which ones are old? Also, why there is one file that weights 12mb? Comparing to the others is very big. For all the countries (except Laos and Thailand that I am downloading now) I have downloaded the same data.

Thanks!

[chrisvwn]

Hi @nreguera In this list the files without -SHPZIP at the end are from version 0.2.4. The upgrade is not yet finalized on github so it seems they did not get upgraded. You can choose to ignore them and in the next version it should pick them up and upgrade them. Or you can choose to reprocess them if you prefer.

The sizes will differ based on the number of entries of the admLevel and the number of stats that have been calculated for the admLevel. In general, the higher the number in the admLevel the more entries and the more stats calculated the bigger the file gets.

[nreguera]

So, if I understood well, if we have the data files with the -SHPZIP at the end, we don´t need the data files without -SHPZIP (in the attached image KHM for example) so we can remove them. And for those that we don´t have yet the -SHPZIP data file (in the attached image THA for example) we have to download a new period and then automatically it will update the data already downloaded in the new -SHPZIP data file format, is that right?

And what about the files containing STRUCT in their name? I have seen that it contains information related to the shape (i.e. area) that is already in the DATA files. Do we need them?

[chrisvwn]

Yes, the files without -SHPZIP were created with a previous version so will not be accessible in the dev version. So if you do not need them e.g. where you have both with and without -SHPZIP you can erase them. That said if it took a long time to create them e.g. for very large countries, you can keep them and they will be upgraded in the next version when the dev is complete.

The files with -STRUCT in the name are needed by the package. They are just to speed up processing. If you delete them they will be recreated the next time you process the country admLevel.

[nreguera]

I just restarted downloading the Thailand rasters, and it started by the first period (2012-04) instead of the last downloaded (2013-02). This is the data that I already have:

data.xlsx

And this is the code I used to restart:

getCtryNlData(
  ctryCode = ctry, # Country
  admLevel = "admin3", # Geograhical level
  nlType = "VIIRS.M", # Periodicity
  nlPeriods = nlRange("201204", "201812"), # timeframe
  ignoreMissing = FALSE, # ignoring missing values
  nlStats = list("sum", na.rm=TRUE)) # aggregation operation for the radiance values

The variable ctry is THA.

Is there anything I should do to use the data that I already downloaded?

[chrisvwn]

@nreguera This is what I meant when I said the files without -SHPZIP at the end will be skipped until the upgrade function is updated for the new version. I will try to fast track a patch that should work for you.

[nreguera]

Oh sorry I didn´t understand it. So it´s ok, I will download them again.